Use the Google Calendar and Google Sheets APIs to select the right recipe on the right day.

Do you also get stressed out when you get the question “what’s for dinner tonight?” You’re not alone. I guess it’s the most asked question as the clock strikes 4 p.m. Deciding what to eat can be a tedious chore. Especially when you have small children with various after-school activities.

To avoid going to the supermarket every day, we usually write up a menu with recipes for the coming week. That way we can buy all our groceries in one supermarket visit. This saves us a lot of time. Besides that, it also saves us money. That is because we are less exposed to all the selling tricks supermarkets use.

Finding recipes for a whole week requires some thinking and planning. We have to take into account the eating preferences of all family members. Besides that, we have a limited time available for cooking each day. To make this easier, I built an automatic meal planner with these features:

• extract the work planning for me and my wife from our shared Google calendars
• extract our preferred recipes from a Google spreadsheet,
• repeat some recipes each week on the same day
• leave one week in between before repeating the other recipes
• I like cooking more than my wife. So on days that I can’t cook the recipes should be short in time
• upload the week menu in a Google calendar

Let’s jump right in.

Using the Google calendar API and Google sheets API

First, we’ll need to create a new Google Cloud project. Before we can use the Google calendar and sheets in this project, we need to enable the API’s. This is very well explained on the web pages below:

• Enabling the Google Calendar API
• Enabling the Google Sheets API

When that’s done, we continue by importing the necessary Python packages.

import config as cfg
import pandas as pd
import numpy as np
from pathlib import Path
from datetime import datetime
from datetime import timedelta
from googleapiclient.discovery import build
from google.oauth2 import service_account

Configuration

For privacy and security reasons, I keep some parameters in a separate config.py file. We import the file with the alias cfg. I will discuss these parameters further below with fictitious values. You can include them for your own app with values relevant to your case.

Scopes

With scopes, we define the access levels for the Google calendars and sheets. We will need read and write access to both the calendars and sheet. Thus we use the URLs below.

SCOPES = ['https://www.googleapis.com/auth/calendar'
          , 'https://www.googleapis.com/auth/spreadsheets']

Google sheet ID and range

SPREADSHEET_ID = <Your Google sheet ID>
RANGE = 'recepten!A:G'

We need to specify the ID of the Google sheet with the recipes. Additionally, we specify the sheet range containing the recipes.

You can find the ID of your Google sheets by right-clicking on the sheet in Google Drive. Then select “Get shareable link”. You can find the ID after “https://drive.google.com/open?id=”.

In my Google sheet “recepten”, columns A to G contain information on each recipe. The screenshot below shows some sample content. So RANGE needs to be set to “recepten!A:G”.

Google Calendar IDs

CALENDARID_1 = <Your Google Calendar ID>
CALENDARID_2 = <Your partner's Google Calendar ID>
CALENDARID_WEEKMENU = <Google Calendar ID for the week menu>

We need to specify the Google Calendar IDs to get the events from. Make sure you have access to all calendars you want to include. You can find the ID by executing this script from the APIs Explorer.

For this project, we will extract the events of only two calendars. But you could adapt the code to loop over more calendars. I’ve also created a separate calendar to upload the recipes.

Event labels

BUSY_EVENTS = [<Labels of busy calendar events>]
FREE_EVENTS = [<Labels of free calendar events>]
ALL_EVENTS = BUSY_EVENTS + FREE_EVENTS

My wife works in shifts and adds them to her Google Calendar by using letter codes. For example: “B” stands for the afternoon shift. This event is one of the BUSY_EVENTS.

When I have a day off, I add “HOLIDAY” to my calendar. This event is one of the FREE_EVENTS.

All the events are full-day events in the Google Calendars. You can use your own event labels scheme.

Traditions

TRADITIONS = {   'Thursday' : 'fries'}

With TRADITIONS, I mean that our family has a few days in the week on which we prepare a certain recipe. As we are from Belgium, this means eating fries once a week (for us on Thursday). And yes, before you’d ask, that is fries with mayonnaise.

You can specify your own traditions in a dictionary, with the name of the day as the key and the recipe as the value.

Number of days to plan ahead

Sometimes we can’t go to the supermarket on the day a new week menu is created. We might need some days to plan ahead. With NB_DAYS_BEFORE we give ourselves some slack. This means that the new week menu will be generated a certain number of days before the previous week menu has finished.

NB_DAYS_BEFORE = 3

Using a service account

We will use a service account to make use of the APIs in the project. The credentials.json file is the file that you can download when enabling the APIs.

We create the credentials creds with the code below. These credentials enable authentication in the Google Calendars and Google sheet.

creds = service_account.Credentials.from_service_account_file("credentials.json", scopes=cfg.SCOPES)

Getting the Google Calendar events

We start by creating the service object with the build method.

service_cal = build('calendar', 'v3', credentials=creds)

We are only interested in the events for the coming week. To filter these events, we specify the dates and format them with isoformat(). The parameters timeMin and timeMax need this format.

def format_date(date):
    date_time = datetime.combine(date, datetime.min.time())
    date_time_utc = date_time.isoformat() + 'Z'
    return date_time_utc

With the method events().list of the service object, we extract the events. The extracted events are then filtered for the BUSY and FREE events. All other events on the Google Calendars are not relevant in this project. We keep the start and end date and the summary of the events.

def get_event_date(event, timepoint):
    return event[timepoint].get('dateTime', event[timepoint].get('date'))

def get_events_by_calendarId(service, calendarId, timeMin, timeMax, allEvents):
    events_result = service.events().list(calendarId=calendarId
                                        , timeMin=timeMin
                                        , timeMax=timeMax
                                        , singleEvents=True
                                        , orderBy='startTime').execute()
    events = events_result.get('items', [])    
    events_list = [(get_event_date(e, 'start'), get_event_date(e, 'end'), e['summary'].upper()) 
                   for e in events 
                   if e['summary'].upper() in allEvents]
    return unfold_events_list(events_list)

Some events spread over more than one day. For example, when you take holidays for more than one day. We unfold these multi-day events in daily events within the range of the coming week.

def unfold_events_list(events_list):
    new_events_list = []
    for e in events_list:
        start = datetime.strptime(e[0], '%Y-%m-%d').date()
        end = datetime.strptime(e[1], '%Y-%m-%d').date()
        delta_days = (end - start).days

        if delta_days > 1:
            for d in range(delta_days):
                unfolded_day = start + timedelta(days=d)
                if unfolded_day >= datetime.now().date() and unfolded_day <= datetime.now().date() + timedelta(days=6):
                    new_events_list.append((unfolded_day, e[2]))
        else:
            new_events_list.append((start, e[2]))
    return new_events_list

Finally, we want a Pandas DataFrame with the events of both calendars for the coming week. To get to that result, we convert the events lists to data frames and merge on the date. We also add the weekday to the merged data frame.

def create_events_df(events_list_1, events_list_2):
    events_df_1 = pd.DataFrame.from_records(events_list_1, columns=['date', 'events_cal_1'])
    events_df_2 = pd.DataFrame.from_records(events_list_2, columns=['date', 'events_cal_2'])
    events_df = events_df_1.merge(events_df_2, on='date', how='outer')
    events_df.date = pd.to_datetime(events_df.date)
    events_df.set_index('date', inplace=True)
    events_df.sort_index(inplace=True)

    dates = list(pd.period_range(START_DAY, NEXT_WEEK, freq='D').values)
    new_idx = []
    for d in dates:
        new_idx.append(np.datetime64(d))

    events_df = events_df.reindex(new_idx)
    events_df.reset_index(inplace=True)
    events_df['weekday'] = events_df.date.apply(lambda x: x.strftime('%A'))
    events_df.set_index('date', inplace=True)
    return events_df

To make sure we cover all dates of the coming week, we use a period_range and reindex the merged data frame.

Getting the recipes from the Google sheet

At this point, we have a data frame with all days of the coming week and the events (if any) occurring in the two calendars. Now we can start to extract the recipes from the Google sheet and assign a recipe to each day. As with the Google Calendar API, let’s start by creating the service object for the Google Sheets API.

service_sheet = build('sheets', 'v4', credentials=creds)

With the method spreadsheets().values().get we can extract the recipes from the Google Sheet.

def get_recipes(service, spreadsheetId, range):
    recipes_result = service.spreadsheets().values().get(spreadsheetId=spreadsheetId, range=range).execute()
    recipes = recipes_result.get('values', [])
    recipes_df = pd.DataFrame.from_records(recipes[1:], columns=recipes[0])
    recipes_df.last_date_on_menu = pd.to_datetime(recipes_df.last_date_on_menu, dayfirst=True)
    recipes_df.set_index('row_number', inplace=True)
    eligible_recipes = recipes_df[ (recipes_df.last_date_on_menu < PREV_WEEK) | (np.isnat(recipes_df.last_date_on_menu)) ]
    return recipes_df, eligible_recipes

Next, we create a data frame with the recipes. I like working with Pandas DataFrames, but you could use other data structures as well of course.

The row_number is a field calculated in the Google Sheet itself. We use the Google Sheet function ROW() for that. It will help to update the field last_date_on_menu in the correct row. We will update that date when a recipe is chosen for the coming week.

We need to make sure that a recipe is only repeated after one week. So we filter recipes_df by last_date_on_menu. This date must be empty or before the previous week.

Generating the week menu

In this step, we will assign an eligible recipe to each day of the coming week.

def generate_weekmenu(service, events_df, traditions, free_events):
    weekmenu_df = events_df.copy()

    for i, r in events_df.iterrows():
        if r.weekday in traditions.keys():
            weekmenu_df.loc[i, 'recipe'] = traditions[r.weekday]
            weekmenu_df.loc[i, 'description'] = ''
        else:
            if r.weekday in ['Saturday', 'Sunday']:
                row_number = choose_recipe('difficult', i, weekmenu_df, eligible_recipes)
                update_sheet(service, row_number, i.strftime('%d-%m-%Y'), cfg.SPREADSHEET_ID)
            elif r.events_cal_1 in free_events or r.events_cal_2 in free_events \
            or pd.isnull(r.events_cal_1) or pd.isnull(r.events_cal_2):
                row_number = choose_recipe('medium', i, weekmenu_df, eligible_recipes)
                update_sheet(service, row_number, i.strftime('%d-%m-%Y'), cfg.SPREADSHEET_ID)
            else:
                row_number = choose_recipe('easy', i, weekmenu_df, eligible_recipes)
                update_sheet(service, row_number, i.strftime('%d-%m-%Y'), cfg.SPREADSHEET_ID)
    return weekmenu_df

To take into account the work planning (BUSY and FREE events), we will use the difficulty of each recipe. A random recipe of the preferred difficulty will be added to weekmenu_df. Finally we drop it from the eligible recipes to avoid duplicate recipes in the same week.

def choose_recipe(difficulty, idx, weekmenu_df, eligible_recipes):
    choice_idx = np.random.choice(eligible_recipes.query("difficulty == '" + difficulty + "'" ).index.values)
    weekmenu_df.loc[idx, 'recipe'] = eligible_recipes.loc[choice_idx, 'recipe']
    weekmenu_df.loc[idx, 'description'] = eligible_recipes.loc[choice_idx, 'description']
    eligible_recipes.drop(choice_idx, inplace=True)
    return choice_idx

The method spreadsheets().values().update updates the Google Sheet.

def update_sheet(service, row_number, date, spreadsheetId):
    range = "recepten!F"  + str(row_number)
    values = [[date]]
    body = {'values' : values}
    result = service.spreadsheets().values().update(spreadsheetId=spreadsheetId
                                                    , range=range
                                                    , valueInputOption='USER_ENTERED'
                                                    , body=body).execute()

We iterate over each row of weekmenu_df. If the weekday is one of the TRADITIONS weekdays, we assign the corresponding recipe. For the other weekdays, we apply the following logic:

• In the weekend, choose a difficult recipe
• During the week, when I’m at home or my wife has a day off, choose a recipe with medium difficulty
• During the week, when I or my wife are at work, choose an easy recipe

Adding the week menu to a Google Calendar

Now that we have a menu for the coming week, we can add it as events to a Google Calendar. I’ve created a separate calendar for it. Share this calendar with the client_email in credentials.json. In the settings of your calendar you also need to give it permission to make changes in the events.

def add_weekmenu_to_calendar(service, weekmenu_df, calendarId):
    for i, r in weekmenu_df.iterrows():
        event = {
        'summary': r.recipe,
        'description': r.description,
        'start': {
            'date': i.date().isoformat(),
            'timeZone': 'Europe/Brussels'
        },
        'end': {
            'date': i.date().isoformat(),
            'timeZone': 'Europe/Brussels'
        }
        }
        event = service.events().insert(calendarId=calendarId, body=event).execute()

Let’s automate

Until now we have taken into account all the requested features for the application. But you would still have to run the code by hand to generate the week menu.

I found this great website PythonAnyWhere where you can schedule Python programs. The free Beginner account allows to schedule one Python program on a daily basis. That’s exactly what we need.

First, we need to stitch all the functions together and put them in a Python file. In this file, I do an extra check to see where we are in the current week menu. I do this by looking at the last date with a recipe in the Google Calendar with get_date_last_event.

def get_date_last_event(service, calendarId):
    events_result = service.events().list(calendarId=calendarId
                                        , singleEvents=True
                                        , orderBy='startTime').execute()
    date_last_event = events_result.get('items', [])[-1]['start']['date'][:10]
    date_last_event = datetime.strptime(date_last_event, '%Y-%m-%d').date()
    return date_last_event

That date is stored in DATE_LAST_RECIPE. If the current day is after DATE_LAST_RECIPE minus NB_DAYS_BEFORE we can generate a new week menu.

You can find the complete script on Github.

if __name__ == '__main__':
    # Getting credentials from credentials.json
    CREDS_PATH = Path.cwd() / "weekmenu" / "credentials.json"
    creds = service_account.Credentials.from_service_account_file(CREDS_PATH, scopes=cfg.SCOPES)

    # Creating service objects
    service_cal = build('calendar', 'v3', credentials=creds)
    service_sheet = build('sheets', 'v4', credentials=creds)

    # Defining dates
    DATE_LAST_RECIPE = get_date_last_event(service_cal, cfg.CALENDARID_WEEKMENU) 
    START_DAY = DATE_LAST_RECIPE + timedelta(days=1)
    NEXT_WEEK = START_DAY + timedelta(days=6)
    PREV_WEEK = START_DAY + timedelta(days=-7)
    START_DAY = format_date(START_DAY)
    NEXT_WEEK = format_date(NEXT_WEEK)
    PREV_WEEK = format_date(PREV_WEEK)

    # Getting the recipes from the Google Sheet
    recipes_df, eligible_recipes = get_recipes(service_sheet, cfg.SPREADSHEET_ID, cfg.RANGE)

    # Check if the last weekmenu is still active
    if DATE_LAST_RECIPE - timedelta(days=cfg.NB_DAYS_BEFORE) < datetime.now().date():
        # Getting the events from the Google Calendars
        events_list_1 = get_events_by_calendarId(service_cal, cfg.CALENDARID_1, START_DAY, NEXT_WEEK, cfg.ALL_EVENTS)
        events_list_2 = get_events_by_calendarId(service_cal, cfg.CALENDARID_2, START_DAY, NEXT_WEEK, cfg.ALL_EVENTS)

        # Merge the two events lists
        events_df = create_events_df(events_list_1, events_list_2)

        # Generating the weekmenu
        weekmenu_df = generate_weekmenu(service_sheet, events_df, cfg.TRADITIONS, cfg.FREE_EVENTS)

        # Adding the weekmenu to a Google Calendar
        add_weekmenu_to_calendar(service_cal, weekmenu_df, cfg.CALENDARID_WEEKMENU)
        print('Week menu is added to Google Calendar')
    else:
        print('Program stopped. Last week menu is not finished yet.')

On PythonAnyWhere I’ve created a subfolder week menu. I’ve uploaded the following files config.py, generate_weekmenu.py and credentials.json.

Project files on PythonAnyWhere.com

I then schedule a daily task that will run the generate_weekmenu.py script in the Tasks section. And voilà, we’re all set.

The result

After the first run of the script, we have a nice menu in our shared Google calendar.

Automated week menu in a shared Google Calendar

Conclusion

This script takes into account your professional schedule on your Google calendars. It selects your preferred recipes from a Google sheet. And by scheduling the script the recipes appear in an automated way in your Google Calendar. This frees you from the annoying chore to decide what to eat.

If you want to take it further, here are some ideas to fine-tune the script:

• take into account the cooking time of a recipe
• allow a tradition of having at least one vegetarian meal per week
• generate a grocery list for the chose recipes

I hope you enjoyed reading this story. If you have questions or suggestions about the script you can write a comment below. And if you liked it, feel free to clap for it.

How touch typing is a universal life hack that will benefit you the rest of your life.

These days we have all heard about the benefits of meditating, eating healthy and going to the gym. There are lots of tricks and hacks people publish to help you get more out of life but I have never come across anything about Touch Typing.

Touch Typing has been a completely overlooked skill that is quite possibly one of the best life hacks anyone can learn without making any radical changes to your life or redefining who you are or anything.

Learning to meditate or going to the gym every day and getting better sleep is for sure great for you, but I’m not sure if it’s even a life hack so I don’t know why Life-Hack lists are always full of them. And for any of those things to be effective you have to learn to actually enjoy doing them, otherwise, it’s just a short-term thing that takes effort and doesn’t change your life in any way at all.

So although learning to Touch Type may sound like perhaps the least sexy life-hack anyone has ever mentioned, I believe it is a truly universal life hack that will pay you back in benefits for the rest of your life, regardless of any personal preferences for pizza or salad, FIFA or yoga etc. Hopefully, this has convinced you enough to go ahead and read the rest of this post to understand why I believe it is so awesome.

Definition

Touch Typing is the ability to type into a computer relying on touch alone. You never need to look down at the keyboard. You can type using purely muscle memory and no longer use ‘active’ thinking to work out how to type letters and words.

This means that, in the long run, you can type much faster without the constraints of looking down at the keys and flicking back up to the screen to see what you typed. This can let you progress onto skills such as being able to transcribe audio as it happens and to make notes super fast, or just be able to write emails in the dark or whilst talking to someone else you can carry on working.

My Background

I have been typing regularly for the past 20 years. I even did a bit of touch typing at school to help learn the keyboard. I type relatively fast and never really thought of it as something I needed to improve. I have one mate who types crazy fast and when he types he looks like a wizard. But, to be honest, the speed that words appear on his computer never really filled me with envy and I assumed my life was just fine.

My typing style involves using most of my fingers so half way to touch typing. I guess I do glance down at the keyboard a lot just to check where keys are. I have definitely reached a plateau at typing speed but thought that was fine.

After a conversation with Alex Denne on my podcast (Growth Mindset Podcast — How to Master Anything), he sent me a link to a touch typing tutorial. It had a great UI, was free and the in-app gamification was cool. Trying just ten minutes a day seemed worth a try.

Benefits

You really don’t know what you don’t know, and I hadn’t realized in the slightest how much my typing style was slowing me down in so many other ways (besides just my words per minute rate).

Fewer Errors

Touch typing means you can be watching the screen the whole time for errors, as such I make about 80% fewer errors in my typing straight away. This is amazing in itself and makes my writing and emailing much better, There is less time needed for checking and less stupid errors getting through which make me appear a complete sloppy dumbass

Free Thought

This is something I really hadn’t expected. But typing with the help of your eyes to check where keys are actual takes a portion of conscious effort dedicated to just the action of typing. By moving the act of typing into the unconscious side of the brain, this frees up your mind to think more openly about other things. This has two great benefits.

When you are no longer worried about typing the immediate words right now you start being able to think about the words ahead. I would say it’s like learning to drive.

Car Driving Analogy

When you first try to drive you only look at the road immediately in front of you and your brain is thinking, “crap I need to move the wheel and change gear and change how much I’m pressing the pedals and Oh God ? there’s someone coming and I need to go round a corner”. And you generally mess it all up and can never plan ahead and you feel like a noob.

Once you learn to master all these different things and automate them in your sub-conscience you start to look much further ahead and see how the road turns. You can already plan ahead what gear you should be in when arriving and your body starts doing things naturally to fit all the adjustments into your plan as you go along rather than reacting to it. You are able to stay aware of potential hazards and enjoy a pleasant conversation or daydreaming or singing along as you drive along just fine. You are not stuck in an endless cycle of focusing only on the immediate road just ahead of you and scrambling to try and do the right thing right now.

It’s the same when you know how to touch type. You can stop thinking about how to put your finger on the right letter at the right time. You don’t need to focus on where a particular letter is on the keyboard and get each word out one by one. You can think about the entire body of thoughts that are in your head and how you are going to distill it into a sensible text

Creativity

When you are no longer writing with a slight disability, you can dedicate your full brain space to the act of thinking about the sentence you want to write. You can also think more long term as you go along about the story you want to tell and genuinely become a better and more creative writer by removing this limitation.

If you have a great idea, you can write it down immediately instead of focusing on how to type the words to express your idea.

Social

This sounds silly, but people often try to talk to you whilst your typing and it can end up in a total mess in two ways, either:

• You stop what you are doing and dedicate your full mental energy to the person. This is very polite but then you might forget what you were doing and have to start again.

• You try to carry on finishing your train of thought whilst keeping the other person on hold in a weird limbo where you give them enough info to keep them there but don’t really tell them anything useful. You finish off your email or idea but it takes longer because you are also talking to the other person and its all a bit rude and both the writing and conversation suck.

With touch typing, you have more brain space available to finish off your typing whilst also listening to the person and answering them in full “real English” sentences, and you can even look at them whilst you do it. Amazing!

Future Thoughts

Increasing accuracy of speech to text and news about projects like Nueralink (brain-machine interface to connect humans and computers) grabbing headlines might make you think there is no point in learning this skill if it will become a relic in the future.

Back to the “Driving analogy”.

People may not consider learning to drive in 2018 on the presumption that driverless cars are coming out in the near future and all cars will be driverless. Learning to drive is expensive, time-consuming, dangerous and quite frankly a little embarrassing. So why bother unless you live somewhere rural and really need to transport yourself right now.

But for touch typing, I do not agree with this argument of wasted time due to future tech making the skill pointless.

Firstly, the investment to learn is much smaller. You can learn touch typing for free! And it doesn’t take half as many hours. And you can’t kill yourself or anyone else whilst doing it. ?

Secondly, there will still be plenty of occasions when the technology isn’t ready or appropriate and traditional “old-school” typing will still apply. Even when once speech-to-text is 100% accurate and people think, hey I can just speak all day and never type, what will actually happen next? We will have to wait 20 years for every office ever to be completely re-designed to let you sit in your own bunker of silence so that you can work with just your computer, instead of working with other people. Except that’s a crap idea and if I’m in an office I want to be able to talk to other people and I really don’t want to hear them talking to their computers all day every day. Every email or document should not be read out loud to the whole office by all my colleagues as they try to decide what to write!

If you’ve ever used Alexa, or Siri or Google Voice etc… then I’m sure you know that feeling when someone starts talking to it and everyone else goes quiet whilst this takes place. It is unnatural and totally not a viable option for real work.

The fact is you will be typing a lot of the time, even if you won’t need to type as much of the time.

And brain-computer interface Neuroprosthetics such as Nueralink despite being super mega awesome fantastic would still take a long time to become practically usable.

You should just embrace the old school version of depositing thoughts from your mind and uploading these thoughts to others via written down text. By learning to touch type and getting familiar with your keyboard, you can make this flow of consciousness as seamless as possible and experience the liberation it provides!

Summary

Touch typing releases brain space by moving conscious thought into the subconscious so you can think more freely about more stuff, and then get that stuff down into writing faster. So you become a more creative and more productive person at the same time.

It doesn’t take long to learn and can be acquired over a month with just 5–10 minutes of active work on it a day. And this time you will quickly get back as you can type and think much faster during your working day.

Not learning to touch type is basically hindering your entire ability to work efficiently if you are a human being who works at all with computers.

So whether you are a writer, CEO, secretary or a student. My number one life hack for 2018 is learning to touch Type.

Links

Touch Typing
I really enjoyed www.typingclub.com as a great site to learn touch typing for free. A good introduction to the keyboard and improving your skills and fun gamification. There are plenty of other websites and apps available.

My Podcast

You can listen to the audio version of this blog here on www.growthmindsetpodcast.com

Or subscribe in your preferred app

• iTunes

• Spotify

• Stitcher

• All other players…

If you liked this article you’ll definitely enjoy these:

• Growth Mindset Podcast — Hacking the world Banks
  Interview with an anonymous Hacker who explains how he hacked into several world banks last year and stole over 45 bitcoins!

• Growth Mindset Podcast — Conversation with a Hitman
  I talk with a real-life hitman. The kind that kills people for a living. It is a mental piece of audio.

• Growth Mindset Podcast — Learnt Helplessness and Learned Optimism
  How to overcome anything that holds you back. Specifically all the things you didn’t even realise were holding you back because they are so ingrained in your mind that they are just an accepted part of life.

If you enjoyed this story, please click the ? button and share to help others find it! Feel free to leave a comment below.

In this article, I’ll show you some exciting facts about American cities, the value of bubble plots in deciding which city to live in, and how to create those plots.

Are you thinking about investing in real estate in 2018? Moving to a new city? When considering these decisions, you need to weigh in different factors like unemployment rate, housing price, the size of the city, safety and so on. Even with all that data and four corresponding bar charts, you’ll still be clueless staring at that table. You’ll try to find the best candidates, but those factors are telling different stories… Sounds like a complex problem.

So, is there a way we can visualize all these factors in 1 chart and compare them ALL? Yes, we can use a bubble plot!

What’s a bubble plot?

A bubble plot is a type of chart that displays more than two dimensions of data (compared to traditional scatter plots). In addition to plotting a dot on an X-Y plane, it uses the size, color, or shape of the point to display more dimensions.

We use unemployment rate as the X-axis, median home price as the Y-axis, and the population of the cities as the size of the dots. This makes a good third dimension. Color is randomly assigned to each city.

The best city in the US to live in is…(wait for it)

Winner: Nashville!

Other recommendations: Austin, Omaha, Milwaukee, Dallas, Minneapolis, Denver and Aurora.

They have low unemployment (and therefore there’s higher chance of finding a job), and low home price, because they are on the lower left side of the plot. What does that mean?

It means you can make your choices based on this plot.

For example, if you consider unemployment rate to be more important and don’t mind the higher home prices, then Honolulu, Oakland, Boston, and San Diego are strong candidates

What about adding safety as another factor?

Sure. Let’s add safety as a fourth factor (the other three factors are still home price, unemployment rate, and population). Instead of randomly assigning a color for a city, we use the color scale for crime (crime rate per 100,000 people). Red means more crime and blue means less.

Does the result change?

It did! If safety is very important for you, then Milwaukee might not be such a great choice among the previous recommendations (even though it’s at the lower left side of the graph).

Now you see the power of a bubble plot: the ability to demonstrate multiple factors in one 2-D plot. If you only have bar charts for those factors, it’s hard for you to identify the cities with an ideal combination of factors. The bubble plot basically created a “visual objective function” for you to optimize a multi-variable decision-making problem.

How do unemployment rate and home price change over time?

We can create an interactive motion chart to add time as a dimension (2013 to 2017) to see how the factors change for these cities over time.

To avoid too much visual information, I didn’t use crime data and used the different colors to represent a few selected cities.

The good news is that the unemployment rate for almost all cities decreased significantly (moving from right to left). But the bad news is that the housing prices are going up pretty fast (especially for San Francisco, San Jose, Los Angles, New York, and Seattle).

Want to create the charts yourself? Here is my code for the bubble plots and the motion chart in R. Have fun playing with the plots :)

################ Bubble Plot ################library(data.table)library(ggplot2)library(ggrepel)

bubble_data <-fread("https://raw.githubusercontent.com/zhendata/Medium_Posts/c007346db1575aca391a6623c87bb5a31a60b365/bubble_plot_merged_city_data.csv",sep=",")

bubble_plot <- ggplot(bubble_data,                aes(x = Unemployment_Rate, y = Home_Price/1000)) +

geom_point(aes(size = Population, fill = Total_Crime),shape=21) +# Create 'Bubble' by assigning size a variable #

scale_fill_continuous(low = "#33FFFF", high ="#FF6699" ) +scale_size_area(max_size = 20)+# Select bubble color scale and bubble maximum size #

geom_text_repel(          aes(label = City),nudge_x = 0,nudge_y = 0.75,size = 6) +# Use geom_text_repel to repel the labels away from each other #

theme_bw()+# Use white background instead of the default grey one #

ggtitle("Best Cities in US to Live in") +labs(x = "Unemployment Rate%", y = "Home Price",       size = "Population",fill="Crime") +theme(plot.title = element_text(size=25, hjust = 0.5),        axis.title=element_text(size=20, face = "bold"),        axis.text=element_text(size=15)) +# Style title and axis #

scale_y_continuous(name="Home Price", breaks = seq(0, 1500, by=250),                       labels=c("0", "250K", "500K", "750K", "1000k",    "1250k", "1500K"))# Make y-axis more readable by replacing scientific number by "K" #

print(bubble_plot)

################# Motion Chart #################library(data.table)library(googleVis)

motion_data <-fread("https://raw.githubusercontent.com/zhendata/Medium_Posts/c007346db1575aca391a6623c87bb5a31a60b365/motion_chart_merged_city_data.csv",sep=",")

motion_chart <- gvisMotionChart(motion_data, idvar = "City", timevar = "Year",xvar = "Unemployment Rate",yvar= "Home Price",sizevar="Population")

plot(motion_chart)# R automatically opens a tab in the browser for you# The flash player needs to be enabled in browser

Click the “⌽” icon to enable Flash

######### Data #########"""The datasets I used are from Zillow (medium housing), FBI’s UCR program, census.gov (population), Bureau of Labor (unemployment). I did some data cleaning and joining for the format I needed in this article, and you can click the links below to download."""bubble_plot_merged_city_data.csv, motion_chart_merged_city_data.csv

Follow me and give me a few claps if you found this helpful!

You can also read my previous articles on data science, real estate, and decision making:

How to analyze seasonality and trends to save money on your apartment lease.
_When I was looking for a new apartment to rent, I started to wonder: is there a data-driven decision making strategy…_medium.freecodecamp.orgHow to Use Data to Predict Rent and Optimize Your Lease Duration So You Can Save Money
_In my last post, we talked about how to pick the best month to sign the lease based on seasonality. Now, how long…_medium.freecodecamp.org

There is more information on the Internet than any human can absorb in a lifetime. What you need is not access to that information, but a scalable way to collect, organize, and analyze it.

You need web scraping.

Web scraping automatically extracts data and presents it in a format you can easily make sense of. In this tutorial, we’ll focus on its applications in the financial market, but web scraping can be used in a wide variety of situations.

If you’re an avid investor, getting closing prices every day can be a pain, especially when the information you need is found across several webpages. We’ll make data extraction easier by building a web scraper to retrieve stock indices automatically from the Internet.

Getting Started

We are going to use Python as our scraping language, together with a simple and powerful library, BeautifulSoup.

• For Mac users, Python is pre-installed in OS X. Open up Terminal and type python --version. You should see your python version is 2.7.x.
• For Windows users, please install Python through the official website.

Next we need to get the BeautifulSoup library using pip, a package management tool for Python.

In the terminal, type:

easy_install pip
pip install BeautifulSoup4

Note: If you fail to execute the above command line, try adding sudo in front of each line.

The Basics

Before we start jumping into the code, let’s understand the basics of HTML and some rules of scraping.

HTML tags
If you already understand HTML tags, feel free to skip this part.

<!DOCTYPE html>  
<html>  
    <head>
    </head>
    <body>
        <h1> First Scraping </h1>
        <p> Hello World </p>
    <body>
</html>

This is the basic syntax of an HTML webpage. Every <tag> serves a block inside the webpage:

1. <!DOCTYPE html>: HTML documents must start with a type declaration.
2. The HTML document is contained between <html> and </html>.
3. The meta and script declaration of the HTML document is between <head>and </head>.
4. The visible part of the HTML document is between <body> and </body>tags.
5. Title headings are defined with the <h1> through <h6> tags.
6. Paragraphs are defined with the <p> tag.

Other useful tags include <a> for hyperlinks, <table> for tables, <tr> for table rows, and <td> for table columns.

Also, HTML tags sometimes come with id or class attributes. The id attribute specifies a unique id for an HTML tag and the value must be unique within the HTML document. The class attribute is used to define equal styles for HTML tags with the same class. We can make use of these ids and classes to help us locate the data we want.

For more information on HTML tags, id and class, please refer to W3Schools Tutorials.

Scraping Rules

1. You should check a website’s Terms and Conditions before you scrape it. Be careful to read the statements about legal use of data. Usually, the data you scrape should not be used for commercial purposes.
2. Do not request data from the website too aggressively with your program (also known as spamming), as this may break the website. Make sure your program behaves in a reasonable manner (i.e. acts like a human). One request for one webpage per second is good practice.
3. The layout of a website may change from time to time, so make sure to revisit the site and rewrite your code as needed

Inspecting the Page

Let’s take one page from the Bloomberg Quote website as an example.

As someone following the stock market, we would like to get the index name (S&P 500) and its price from this page. First, right-click and open your browser’s inspector to inspect the webpage.

Try hovering your cursor on the price and you should be able to see a blue box surrounding it. If you click it, the related HTML will be selected in the browser console.

From the result, we can see that the price is inside a few levels of HTML tags, which is <div class="basic-quote"> → <div class="price-container up">→ <div class="price">.

Similarly, if you hover and click the name “S&P 500 Index”, it is inside <div class="basic-quote"> and <h1 class="name">.

Now we know the unique location of our data with the help of class tags.

Jump into the Code

Now that we know where our data is, we can start coding our web scraper. Open your text editor now!

First, we need to import all the libraries that we are going to use.

# import libraries
import urllib2
from bs4 import BeautifulSoup

Next, declare a variable for the url of the page.

# specify the url
quote_page = ‘http://www.bloomberg.com/quote/SPX:IND'

Then, make use of the Python urllib2 to get the HTML page of the url declared.

# query the website and return the html to the variable ‘page’
page = urllib2.urlopen(quote_page)

Finally, parse the page into BeautifulSoup format so we can use BeautifulSoup to work on it.

# parse the html using beautiful soup and store in variable `soup`
soup = BeautifulSoup(page, ‘html.parser’)

Now we have a variable, soup, containing the HTML of the page. Here’s where we can start coding the part that extracts the data.

Remember the unique layers of our data? BeautifulSoup can help us get into these layers and extract the content with find(). In this case, since the HTML class name is unique on this page, we can simply query <div class="name">.

# Take out the <div> of name and get its value
name_box = soup.find(‘h1’, attrs={‘class’: ‘name’})

After we have the tag, we can get the data by getting its text.

name = name_box.text.strip() # strip() is used to remove starting and trailing
print name

Similarly, we can get the price too.

# get the index price
price_box = soup.find(‘div’, attrs={‘class’:’price’})
price = price_box.text
print price

When you run the program, you should be able to see that it prints out the current price of the S&P 500 Index.

Export to Excel CSV

Now that we have the data, it is time to save it. The Excel Comma Separated Format is a nice choice. It can be opened in Excel so you can see the data and process it easily.

But first, we have to import the Python csv module and the datetime module to get the record date. Insert these lines to your code in the import section.

import csv
from datetime import datetime

At the bottom of your code, add the code for writing data to a csv file.

# open a csv file with append, so old data will not be erased
with open(‘index.csv’, ‘a’) as csv_file:
 writer = csv.writer(csv_file)
 writer.writerow([name, price, datetime.now()])

Now if you run your program, you should able to export an index.csv file, which you can then open with Excel, where you should see a line of data.

So if you run this program everyday, you will be able to easily get the S&P 500 Index price without rummaging through the website!

Going Further (Advanced uses)

Multiple Indices
So scraping one index is not enough for you, right? We can try to extract multiple indices at the same time.

First, modify the quote_page into an array of URLs.

quote_page = [‘http://www.bloomberg.com/quote/SPX:IND', ‘http://www.bloomberg.com/quote/CCMP:IND']

Then we change the data extraction code into a for loop, which will process the URLs one by one and store all the data into a variable data in tuples.

# for loop
data = []
for pg in quote_page:
 # query the website and return the html to the variable ‘page’
 page = urllib2.urlopen(pg)

# parse the html using beautiful soap and store in variable `soup`
 soup = BeautifulSoup(page, ‘html.parser’)

# Take out the <div> of name and get its value
 name_box = soup.find(‘h1’, attrs={‘class’: ‘name’})
 name = name_box.text.strip() # strip() is used to remove starting and trailing

# get the index price
 price_box = soup.find(‘div’, attrs={‘class’:’price’})
 price = price_box.text

# save the data in tuple
 data.append((name, price))

Also, modify the saving section to save data row by row.

# open a csv file with append, so old data will not be erased
with open(‘index.csv’, ‘a’) as csv_file:
 writer = csv.writer(csv_file)
 # The for loop
 for name, price in data:
 writer.writerow([name, price, datetime.now()])

Rerun the program and you should be able to extract two indices at the same time!

Advanced Scraping Techniques

BeautifulSoup is simple and great for small-scale web scraping. But if you are interested in scraping data at a larger scale, you should consider using these other alternatives:

1. Scrapy, a powerful python scraping framework
2. Try to integrate your code with some public APIs. The efficiency of data retrieval is much higher than scraping webpages. For example, take a look at Facebook Graph API, which can help you get hidden data which is not shown on Facebook webpages.
3. Consider using a database backend like MySQL to store your data when it gets too large.

Adopt the DRY Method

DRY stands for “Don’t Repeat Yourself”, try to automate your everyday tasks like this person. Some other fun projects to consider might be keeping track of your Facebook friends’ active time (with their consent of course), or grabbing a list of topics in a forum and trying out natural language processing (which is a hot topic for Artificial Intelligence right now)!

If you have any questions, please feel free to leave a comment below.

References
http://www.gregreda.com/2013/03/03/web-scraping-101-with-python/
http://www.analyticsvidhya.com/blog/2015/10/beginner-guide-web-scraping-beautiful-soup-python/

This article was originally published on Altitude Labs’ blog and was written by our software engineer, Leonard Mok. Altitude Labs is a software agency that specializes in personalized, mobile-first React apps.