This is especially vital in finance and trading, where clear visuals can uncover risks, opportunities, and patterns, directly affecting decisions on position sizing, timing, and confidence.

Today, we'll use FMP APIs to interpret earnings data: extracting announcements, surprises, and price reactions across almost 1,000 stocks to identify actionable patterns in post‑earnings movements.

Here’s exactly what we’ll build:

• Sector heatmap: Maps strongest 3/10-day post-earnings reactions by sector/market-cap buckets.

• EPS scatter: Tests if earnings beats drive returns (sector-colored, with regression).

• Return violins: Shows 3-day post-earnings volatility/skew by sector and market-cap.

• Mega-tech time series: Tracks AAPL/MSFT/NVDA post-earnings patterns over time.

• Monthly seasonality: Reveals calendar edges in post-earnings returns/surprises.

• Regime cross-section: Tests sector robustness across bull/bear/sideways markets.

What we'll cover:

1. Prerequisites

2. Data Extraction

3. Storytelling with Charts and Visuals

   • Sector Heatmap

   • Mega‑Cap Tech Time Series

   • EPS Surprise Scatter Plot

   • Return Distribution Violins

   • Monthly Seasonality

   • Regime Cross-Section

4. What Did We Get Out of All This Storyline?

5. Final Thoughts

Prerequisites

To follow along, you should be comfortable with Python and basic data manipulation in pandas.

This is a code-first guide. I’ll focus on the workflow and the story the charts reveal, and I won’t explain every line of Python. You should be comfortable reading pandas code, loops, and basic plotting logic so you can follow along without needing a step-by-step breakdown of each block.

You’ll need:

• Python 3.10+

• A Financial Modeling Prep (FMP) API key

• pandas, numpy, matplotlib, seaborn, scipy installed

• Enough local compute and patience to run API loops across a large stock universe

Data Extraction

In the first part of this article, we need to collect all the data required for our visualisation exercise. Using FMP’s Stock Screener API, we will retrieve NASDAQ stocks. The first API call will return 1,000 stocks.

import requests
import pandas as pd
import numpy as np
import json
from datetime import datetime, timedelta
import seaborn as sns
import matplotlib.pyplot as plt
from scipy import stats

token = 'YOUR FMP TOKEN'

url = f'https://financialmodelingprep.com/stable/company-screener'
querystring = {"apikey":token,"country":"US", "exchange": "NASDAQ", "isActiveTrading": True, "isEtf": False, "isFund": False}
resp = requests.get(url, querystring).json()

df_universe = pd.DataFrame(resp)
df_universe = df_universe[df_universe['exchangeShortName'] == 'NASDAQ']
df_universe

This will give us 1,000 stocks! Next, we'll bin the market capitalisation to gain a better understanding of the results later on, and we will keep only four columns that are necessary: the symbol, name, market cap, and sector.

bins = [0,
        250_000_000,    # 250M
        2_000_000_000,  # 2B
        10_000_000_000, # 10B
        200_000_000_000,# 200B
        float("inf")]

labels = ["Micro", "Small", "Mid", "Large", "Mega"]

df_universe["marketCap"] = pd.cut(df_universe["marketCap"], bins=bins, labels=labels, right=False)
df_universe = df_universe[['symbol', 'companyName', 'marketCap', 'sector']]
df_universe

Now it is time to retrieve the earnings using FMP’s Earnings Report API. We'll loop through each symbol and collect all the earnings the endpoint provides to us.

symbols = df_universe['symbol'].to_list()

all_dfs = []

for symbol in symbols:
    url = f"https://financialmodelingprep.com/stable/earnings?symbol={symbol}"
    params = {"apikey": token}
    resp = requests.get(url, params=params)

    if resp.status_code != 200:
        print(f"Error for {symbol}: {resp.status_code} - {resp.text}")
        continue

    data = resp.json()
    if not data:
        print(f"No data for {symbol}")
        continue

    df_symbol = pd.DataFrame(data)
    df_symbol["symbol"] = symbol
    all_dfs.append(df_symbol)

# Single DataFrame with all earnings
df_earnings = pd.concat(all_dfs, ignore_index=True)
df_earnings = df_earnings.dropna(subset=['epsActual', 'epsEstimated', 'revenueActual','revenueEstimated'])
df_earnings

Now we'll calculate the surprise, both for earnings and revenue in percentage terms, so we can later compare apples with apples! We'll keep everything from 2010 onwards.

df_earnings["eps_surprise"] = ((df_earnings["epsActual"] - df_earnings["epsEstimated"]) /
                               abs(df_earnings["epsEstimated"]) * 100).round(2)

df_earnings["revenue_surprise"] = ((df_earnings["revenueActual"] - df_earnings["revenueEstimated"]) /
                                   abs(df_earnings["revenueEstimated"]) * 100).round(2)

df_earnings = df_earnings[['symbol', 'date', 'eps_surprise', 'revenue_surprise']]

df_earnings["date"] = pd.to_datetime(df_earnings["date"])
df_earnings = df_earnings[df_earnings["date"] > "2009-12-31"]

Lastly, as a final step in gathering the data needed for visualization, using FMP’s Historical Index Full Chart API, we'll loop through the stocks in our dataframe, retrieve the historical daily prices, and calculate the return of the stock 3 and 10 trading days before and after the earnings announcement.

unique_symbols = df_earnings["symbol"].unique()

price_results = []

print(f"Processing {len(unique_symbols)} symbols...")

for symbol in unique_symbols:
    # Fetch full historical prices
    url = f"https://financialmodelingprep.com/stable/historical-price-eod/full"
    params = {"apikey":token, "symbol":symbol, "from":'2009-10-01'}
    resp = requests.get(url, params=params)

    if resp.status_code != 200:
        print(f"Error for {symbol}: {resp.status_code}")
        continue

    data = resp.json()

    hist_df = pd.DataFrame(data)
    hist_df["date"] = pd.to_datetime(hist_df["date"])
    hist_df = hist_df.sort_values("date").reset_index(drop=True)

    # Get matching earnings rows
    earnings_symbol = df_earnings[df_earnings["symbol"] == symbol].copy()

    for _, row in earnings_symbol.iterrows():
        earn_date = pd.to_datetime(row["date"]).date()

        # === 3-DAY WINDOWS ===
        pre3_mask = (hist_df["date"].dt.date < earn_date) & \
                    (hist_df["date"].dt.date >= earn_date - timedelta(days=10))
        pre3 = hist_df[pre3_mask].tail(3)

        post3_mask = (hist_df["date"].dt.date > earn_date) & \
                     (hist_df["date"].dt.date <= earn_date + timedelta(days=10))
        post3 = hist_df[post3_mask].head(3)

        pre3_start = pre3["close"].iloc[0] if len(pre3) >= 3 else None
        pre3_end = pre3["close"].iloc[-1] if len(pre3) >= 1 else None
        post3_end = post3["close"].iloc[-1] if len(post3) >= 3 else None

        pct_pre_3d = ((pre3_end - pre3_start) / pre3_start * 100) if pre3_start and pre3_end else None
        pct_post_3d = ((post3_end - pre3_end) / pre3_end * 100) if pre3_end and post3_end else None

        # === 10-DAY WINDOWS ===
        pre10_mask = (hist_df["date"].dt.date < earn_date) & \
                     (hist_df["date"].dt.date >= earn_date - timedelta(days=20))
        pre10 = hist_df[pre10_mask].tail(10)

        post10_mask = (hist_df["date"].dt.date > earn_date) & \
                      (hist_df["date"].dt.date <= earn_date + timedelta(days=20))
        post10 = hist_df[post10_mask].head(10)

        pre10_start = pre10["close"].iloc[0] if len(pre10) >= 10 else None
        pre10_end = pre10["close"].iloc[-1] if len(pre10) >= 1 else None
        post10_end = post10["close"].iloc[-1] if len(post10) >= 10 else None

        pct_pre_10d = ((pre10_end - pre10_start) / pre10_start * 100) if pre10_start and pre10_end else None
        pct_post_10d = ((post10_end - pre10_end) / pre10_end * 100) if pre10_end and post10_end else None

        price_results.append({
            "symbol": symbol,
            "earn_date": earn_date,
            "month": earn_date.month,
            "pct_pre_3d": round(pct_pre_3d, 2) if pct_pre_3d else None,
            "pct_post_3d": round(pct_post_3d, 2) if pct_post_3d else None,
            "pct_pre_10d": round(pct_pre_10d, 2) if pct_pre_10d else None,
            "pct_post_10d": round(pct_post_10d, 2) if pct_post_10d else None,
            "eps_surprise": row["eps_surprise"],
            "revenue_surprise": row["revenue_surprise"]
        })



df_earnings = pd.DataFrame(price_results)
df_earnings.dropna(inplace=True)
df_earnings = df_universe.merge(df_earnings, on="symbol")
df_earnings

As you can see, at the end of the code, we have also merged the initial dataset, so all the information, such as name, marketCap, and sector, is now in a single dataset.

Storytelling with Charts and Visuals

Sector Heatmap

First, we'll present the Sector Heatmap of average 3-day post-earnings returns segmented by sector and market-cap category. This basic visualisation highlights areas with the most significant reactions, enabling traders to swiftly identify high-alpha sectors and market caps for earnings strategies.

# Aggregate: average post-earnings returns and EPS surprise
agg = (
    df_earnings
    .dropna(subset=['pct_post_3d', 'pct_post_10d', 'eps_surprise', 'marketCap', 'sector'])
    .groupby(['sector', 'marketCap'])
    .agg(
        avg_post3d=('pct_post_3d', 'mean'),
        avg_post10d=('pct_post_10d', 'mean'),
        avg_eps_surprise=('eps_surprise', 'mean')
    )
    .reset_index()
)

# Heatmap: average 3-day post-earnings return
heatmap_3d = agg.pivot(index='sector', columns='marketCap', values='avg_post3d')

plt.figure(figsize=(12, 8))
sns.heatmap(
    heatmap_3d,
    annot=True,
    fmt='.2f',
    cmap='RdYlGn',
    center=0,
    linewidths=0.5,
    linecolor='grey'
)
plt.title('Average 3-Day Post-Earnings Return by Sector and Market-Cap Bucket')
plt.xlabel('Market-cap bucket')
plt.ylabel('Sector')
plt.xticks(rotation=45, ha='right')
plt.tight_layout()
plt.show()

Consumer Cyclical and Materials are performing really well, with small and mid caps seeing positive reactions over 1.1%. Real Estate is also doing great, jumping up to +4.0% in mid caps. Energy and Financials are holding steady, staying close to zero. Technology, on the other hand, is showing more muted gains, under 1.1%, indicating there might be limited immediate upside from the big tech earnings.

Building on the 3‑day heatmap, we'll now look at the Sector Heatmap for average 10‑day post‑earnings returns by sector and market‑cap category. This extends the timeframe to capture momentum persistence, revealing which sectors maintain or reverse short‑term reactions.

# Heatmap: average 10-day post-earnings return
heatmap_10d = agg.pivot(index='sector', columns='marketCap', values='avg_post10d')

plt.figure(figsize=(12, 8))
sns.heatmap(
    heatmap_10d,
    annot=True,
    fmt='.2f',
    cmap='RdYlGn',
    center=0,
    linewidths=0.5,
    linecolor='grey'
)
plt.title('Average 10-Day Post-Earnings Return by Sector and Market-Cap Bucket')
plt.xlabel('Market-cap bucket')
plt.ylabel('Sector')
plt.xticks(rotation=45, ha='right')
plt.tight_layout()
plt.show()

Consumer Cyclical stands out with peaks at 3.2% (mega caps), and Industrials and Health Care show consistent gains in mid and large caps around 1.1%. Real Estate has eased after its 3-day surge. Technology has seen a small boost in mega caps (+1.8%) but remains less active overall compared to cyclicals.

Mega‑Cap Tech Time Series

Extending the heatmaps, we’ll now look at a Mega-Cap Tech time series. It tracks 10-day post-earnings returns over time for AAPL, MSFT, NVDA, and a few other mega-cap tech names.

A bubble chart works well here because it encodes more than one thing at once. The x-axis is the earnings date, the y-axis is the 10-day post-earnings return, the bubble size scales with the absolute EPS surprise magnitude, and the color shows whether the surprise was a beat or a miss. This makes it easy to spot outlier quarters and see whether big surprises consistently lead to bigger post-earnings moves.

# Define mega-cap tech tickers (top ones from data: AAPL, MSFT, NVDA, AMZN, GOOG/GOOGL, META)
tech_tickers = ['AAPL', 'MSFT', 'NVDA', 'AMZN', 'GOOG', 'GOOGL', 'META']

# Filter data for mega-cap tech
df_tech = (
    df_earnings[df_earnings['symbol'].isin(tech_tickers)]
    .dropna(subset=['earn_date', 'pct_post_10d', 'eps_surprise'])
    .sort_values('earn_date')
    .assign(
        earn_date=lambda x: pd.to_datetime(x['earn_date'])
    )
)

# Create time-series plot: pct_post_10d vs earn_date, sized/color by eps_surprise
plt.figure(figsize=(14, 8))

# Scatter plot
scatter = plt.scatter(
    df_tech['earn_date'],
    df_tech['pct_post_10d'],
    s=np.abs(df_tech['eps_surprise']) * 50 + 20,  # Size by abs(eps_surprise)
    c=df_tech['eps_surprise'],
    cmap='RdYlBu_r',
    alpha=0.7,
    edgecolors='black',
    linewidth=0.5
)

plt.colorbar(scatter, label='EPS Surprise (%)')
plt.xlabel('Earnings Date')
plt.ylabel('10-Day Post-Earnings Return (%)')
plt.title('Mega-Cap Tech: 10-Day Post-Earnings Returns vs Time\n(Point size/color by EPS Surprise)')
plt.grid(True, alpha=0.3)

# Add trend line
z = np.polyfit(pd.to_numeric(df_tech['earn_date']), df_tech['pct_post_10d'], 1)
p = np.poly1d(z)
plt.plot(df_tech['earn_date'], p(pd.to_numeric(df_tech['earn_date'])), "r--", alpha=0.8, linewidth=2, label=f'Trend: {z[0]:.3f}x + {z[1]:.1f}')

plt.legend()
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

That large red bubble around 2018 is almost certainly AAPL’s Q4 2018 earnings miss (Jan 2019 announcement, but fiscal Q4 2018 data) and it stands out because:

• Large size = massive EPS surprise magnitude (Apple cut guidance dramatically, ~10% miss)

• Red colour = negative surprise

• Low Y position = poor 10‑day return (~-10% range visible)

This was Apple’s infamous “iPhone demand warning” that triggered the January 2019 market panic. Perfect example of how one outlier event can anchor the whole trend line downward in your visualisation.

EPS Surprise Scatter Plot

After identifying major tech trends, let's now look at the EPS Surprise Scatter plots. This plot checks a simple hypothesis. Do earnings beats lead to positive returns, and do misses lead to negative returns? We plot EPS surprise on the x-axis and post-earnings returns on the y-axis, then add a regression line to show the average relationship.

# Prepare data: drop NaNs and convert earn_date if needed (not used here)
df_plot = (
    df_earnings
    .dropna(subset=['eps_surprise', 'pct_post_3d', 'pct_post_10d', 'sector'])
    .copy()
)

# 1. Scatter: EPS Surprise vs 3-Day Post-Return, colored by sector
plt.figure(figsize=(12, 5))

plt.subplot(1, 2, 1)
sns.scatterplot(
    data=df_plot,
    x='eps_surprise',
    y='pct_post_3d',
    hue='sector',
    alpha=0.6,
    s=40
)

# Regression line (overall)
slope, intercept, r_value, p_value, std_err = stats.linregress(df_plot['eps_surprise'], df_plot['pct_post_3d'])
line = slope * df_plot['eps_surprise'] + intercept
plt.plot(df_plot['eps_surprise'], line, 'red', linestyle='--', linewidth=2,
         label=f'y = {slope:.3f}x + {intercept:.2f}\nR²={r_value**2:.3f}')
plt.xlabel('EPS Surprise (%)')
plt.ylabel('3-Day Post-Earnings Return (%)')
plt.title('EPS Surprise vs 3-Day Post-Return by Sector')
plt.legend(bbox_to_anchor=(1.05, 1), loc='upper left')
plt.grid(True, alpha=0.3)

# 2. Scatter: EPS Surprise vs 10-Day Post-Return, colored by sector
plt.subplot(1, 2, 2)
sns.scatterplot(
    data=df_plot,
    x='eps_surprise',
    y='pct_post_10d',
    hue='sector',
    alpha=0.6,
    s=40
)

# Regression line (overall)
slope10, intercept10, r_value10, p_value10, std_err10 = stats.linregress(df_plot['eps_surprise'], df_plot['pct_post_10d'])
line10 = slope10 * df_plot['eps_surprise'] + intercept10
plt.plot(df_plot['eps_surprise'], line10, 'red', linestyle='--', linewidth=2,
         label=f'y = {slope10:.3f}x + {intercept10:.2f}\nR²={r_value10**2:.3f}')
plt.xlabel('EPS Surprise (%)')
plt.ylabel('10-Day Post-Earnings Return (%)')
plt.title('EPS Surprise vs 10-Day Post-Return by Sector')
plt.legend(bbox_to_anchor=(1.05, 1), loc='upper left')
plt.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

# Optional: Summary table of correlations by sector
corr_3d = df_plot.groupby('sector')[['eps_surprise', 'pct_post_3d']].corr().unstack().xs('pct_post_3d', level=1, axis=1)['eps_surprise']
corr_10d = df_plot.groupby('sector')[['eps_surprise', 'pct_post_10d']].corr().unstack().xs('pct_post_10d', level=1, axis=1)['eps_surprise']

corr_df = pd.DataFrame({
    'Corr_EPS_3Day': corr_3d.round(3),
    'Corr_EPS_10Day': corr_10d.round(3)
}).sort_values('Corr_EPS_10Day', ascending=False)

The red dashed trend line illustrates the typical relationship: for every 1% EPS beat, stocks tend to gain about 0.05–0.1% over 3 to 10 days. The gentle slope suggests that while surprises can give a little boost, they don’t guarantee large moves.

You’ll notice that Consumer Cyclical dots mainly cluster in the upper right (beats leading to gains), and Real Estate shows a steeper increase. The wide spread around the line indicates that other factors often influence stock movements beyond surprises.

Return Distribution Violins

Heatmaps show averages, but averages can hide risk. Violin plots show the full distribution of returns, including how wide the outcomes are and whether the tails are heavy. Here we plot 3-day post-earnings return distributions by sector and by market-cap bucket.

# Prepare data
df_plot = (
    df_earnings
    .dropna(subset=['pct_post_3d', 'sector', 'marketCap'])
    .copy()
)

# 1. Violin plot: 3-day post-returns by sector
plt.figure(figsize=(15, 6))

plt.subplot(1, 2, 1)
sns.violinplot(
    data=df_plot,
    x='sector',
    y='pct_post_3d',
    inner='quartile',
    palette='Set2'
)
plt.title('Distribution of 3-Day Post-Earnings Returns by Sector (Violin)')
plt.xlabel('Sector')
plt.ylabel('3-Day Post-Earnings Return (%)')
plt.xticks(rotation=45, ha='right')
plt.grid(True, alpha=0.3)

# 2. Violin plot: 3-day post-returns by market-cap group
plt.subplot(1, 2, 2)
sns.violinplot(
    data=df_plot,
    x='marketCap',
    y='pct_post_3d',
    inner='quartile',
    palette='Set3'
)
plt.title('Distribution of 3-Day Post-Earnings Returns by Market-Cap (Violin)')
plt.xlabel('Market-cap bucket')
plt.ylabel('3-Day Post-Earnings Return (%)')
plt.xticks(rotation=45, ha='right')
plt.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()


plt.show()

# Summary statistics table
summary = df_plot.groupby(['sector', 'marketCap'])['pct_post_3d'].agg(['mean', 'median', 'std', 'count']).round(2)
print("Summary Statistics: Mean/Median/Std/Count of 3-Day Returns by Sector & Market-Cap")
print(summary)

All violins concentrate near zero with modest variations (±5%), indicating that post-earnings reactions are generally noisy and lack a clear direction. Markets efficiently incorporate expectations, resulting in little predictable advantage. Consumer Cyclical and Materials sectors display slightly more frequent upside surprises, while small caps exhibit the greatest variability, reflecting higher risk and occasional gains. Not every visualization reveals alpha; this one honestly illustrates the difficulty involved.

Monthly Seasonality

After observing narrow return distributions near zero, let's now look at Monthly Seasonality in four panels: average 3/10‑day post‑returns, EPS surprises, and event counts by month. This reveals calendar effects,  systematic seasonal biases ,  that can influence timing of entries despite noisy individual responses.

# 1. Ensure earn_date is datetime
df_month = (
    df_earnings
    .dropna(subset=['earn_date', 'pct_post_3d', 'pct_post_10d', 'eps_surprise'])
    .copy()
)

df_month['earn_date'] = pd.to_datetime(df_month['earn_date'])

# 2. Derive month number and name
df_month['month_num'] = df_month['earn_date'].dt.month
df_month['month_name'] = df_month['earn_date'].dt.strftime('%b')

# 3. Aggregate averages by month
monthly_agg = (
    df_month
    .groupby('month_num')
    .agg(
        pct_post_3d_mean=('pct_post_3d', 'mean'),
        pct_post_10d_mean=('pct_post_10d', 'mean'),
        eps_surprise_mean=('eps_surprise', 'mean'),
        n_obs=('earn_date', 'count')
    )
    .reset_index()
    .sort_values('month_num')
)

# Keep a stable month order and names
month_order = monthly_agg['month_num'].tolist()
month_labels = df_month.drop_duplicates('month_num').set_index('month_num')['month_name'].reindex(month_order)

monthly_agg['month_name'] = month_labels.values

# 4. Plot bar charts
fig, axes = plt.subplots(2, 2, figsize=(14, 10))
fig.suptitle('Monthly Seasonality of Post-Earnings Returns and EPS Surprise', fontsize=16)

# Avg 3-day return
axes[0, 0].bar(monthly_agg['month_name'], monthly_agg['pct_post_3d_mean'], color='skyblue')
axes[0, 0].set_title('Avg 3-Day Post-Earnings Return by Month')
axes[0, 0].set_ylabel('Return (%)')
axes[0, 0].grid(alpha=0.3)

# Avg 10-day return
axes[0, 1].bar(monthly_agg['month_name'], monthly_agg['pct_post_10d_mean'], color='lightgreen')
axes[0, 1].set_title('Avg 10-Day Post-Earnings Return by Month')
axes[0, 1].set_ylabel('Return (%)')
axes[0, 1].grid(alpha=0.3)

# Avg EPS surprise
axes[1, 0].bar(monthly_agg['month_name'], monthly_agg['eps_surprise_mean'], color='salmon')
axes[1, 0].set_title('Avg EPS Surprise by Month')
axes[1, 0].set_ylabel('EPS Surprise')
axes[1, 0].grid(alpha=0.3)

# Number of observations
axes[1, 1].bar(monthly_agg['month_name'], monthly_agg['n_obs'], color='gold')
axes[1, 1].set_title('Number of Earnings Events by Month')
axes[1, 1].set_ylabel('Count')
axes[1, 1].grid(alpha=0.3)

for ax in axes.ravel():
    ax.set_xlabel('Month')
    ax.tick_params(axis='x', rotation=0)

plt.tight_layout()
plt.show()

Jan/Oct tend to have the best 3‑day returns, about 0.8%, while May/Jul usually see weaker results. The 10‑day trends show a similar but gentler pattern, with February and August reaching peaks. EPS surprises are slightly negative in January and May, possibly due to tough comparisons, and there are fewer events in July, August, and December because of holidays. While there’s a hint of seasonality, its impact is quite small, around 0.5%.

Regime Cross-Section

Finally, after subtle monthly patterns, we'll look at the Regime Cross‑Section: sector 10‑day post‑earnings returns by market regime (heatmap at the top, bars below). This stress‑tests earlier findings  ( do patterns persist across bull, bear, and COVID eras), revealing rotation opportunities and regime dependence.

# Prepare data with year extraction
df_regimes = (
    df_earnings
    .dropna(subset=['earn_date', 'pct_post_10d', 'sector'])
    .copy()
)

df_regimes['earn_date'] = pd.to_datetime(df_regimes['earn_date'])
df_regimes['year'] = df_regimes['earn_date'].dt.year

# Define market regimes (adjust years based on your data/market history)
# Example: Bull (2023-2025), Bear/Transition (2022), COVID (2020-2021), etc.
def assign_regime(year):
    if year >= 2023:
        return 'Bull (2023+)'
    elif year == 2022:
        return 'Bear (2022)'
    elif 2020 <= year <= 2021:
        return 'COVID Recovery'
    elif 2018 <= year <= 2019:
        return 'Pre-COVID'
    else:
        return 'Earlier'

df_regimes['market_regime'] = df_regimes['year'].apply(assign_regime)

# 1. Aggregate: average 10-day returns by sector and regime/year
agg_data = (
    df_regimes
    .groupby(['sector', 'market_regime'])['pct_post_10d']
    .agg(['mean', 'count'])
    .reset_index()
    .query('count >= 5')  # Filter low-sample regimes
)

# 2. Visualization: Heatmap first (quick overview)
plt.figure(figsize=(12, 8))

plt.subplot(2, 1, 1)
pivot_heatmap = agg_data.pivot(index='sector', columns='market_regime', values='mean')
sns.heatmap(pivot_heatmap, annot=True, fmt='.2f', cmap='RdYlGn', center=0, linewidths=0.5)
plt.title('Average 10-Day Post-Earnings Returns: Sector x Market Regime Heatmap')

# 3. Bar charts: By regime (stacked by sector)
plt.subplot(2, 1, 2)
regime_order = agg_data.groupby('market_regime')['mean'].mean().sort_values(ascending=False).index
sns.barplot(data=agg_data, x='market_regime', y='mean', hue='sector',
            palette='Set2', order=regime_order)
plt.title('Average 10-Day Returns by Market Regime (Colored by Sector)')
plt.ylabel('10-Day Post-Return (%)')
plt.xlabel('Market Regime')
plt.xticks(rotation=45, ha='right')
plt.legend(bbox_to_anchor=(1.05, 1), loc='upper left')
plt.grid(axis='y', alpha=0.3)

plt.tight_layout()
plt.show()

# 5. Summary tables
print("Average Returns by Sector x Market Regime (min 5 obs):")
print(agg_data.pivot(index='sector', columns='market_regime', values='mean').round(2))

# 6. Ranking: Best/worst performing sectors by regime
print("\nTop/Bottom Sectors by Regime:")
for regime in regime_order:
    regime_data = agg_data[agg_data['market_regime'] == regime].sort_values('mean', ascending=False)
    print(f"\n{regime}:")
    print(regime_data[['sector', 'mean', 'count']].round(2).head(3))

Consumer Cyclical does well during Bull (2023+) and COVID Recovery (1.5–2%), but it’s less favorable in Bear 2022. Utilities turned negative before COVID. The bottom bars show the COVID era led overall gains (1%), with Basic Materials and Industrials being the strongest. The recent Bull remains positive but less so. Sector leadership shifts depending on the market regime , there are no consistent winners.

What Did We Get Out of All This Storyline?

Guiding you through six interconnected visualizations, we’ve turned 15 years of earnings data into a clear and engaging story.

Each chart responds to a specific question, yet together, they paint a bigger picture: earnings surprises influence markets, but not in the same way everywhere. Some sectors, periods, and regimes often provide consistent advantages, while others don’t.

Here’s what the data shows us:

• No definitive alpha here, but specific opportunities are present: Markets are mostly efficient,  returns hover near zero with weak surprise correlations ,  yet Consumer Cyclicals and Materials consistently show upside potential across different timeframes and market sizes. Timing your sector choice is important.

• Timing windows alter the story: 3-day reactions benefit Real Estate mid-caps (+4%), while 10-day reactions shift leadership to Consumer Cyclical mega-caps (+3.2%). Don’t assume all earnings reactions occur at the same pace.

• Mega-tech hype isn’t eternal: The bubble chart shows AAPL/MSFT/NVDA delivered strong returns from 2020–2022, but the falling trend since then indicates waning market enthusiasm. Don’t chase yesterday’s overhyped stocks.

• Calendar patterns reward patience: January and October deliver slightly stronger post-earnings returns (~0.8%), while July and August tend to have lower liquidity. Combine seasonal timing with sector choices for additional gains.

• Market regimes change winners: Cyclicals underperformed during COVID recovery and the bull run (2023+), while Industrials peaked during the recovery. There are no universal “best performers,” only the best performers for now. Adjust to the regime.

• The actionable setup: Small to mid-cap cyclical longs in January during bull markets combine all these signals for maximum conviction ,  where sector timing, seasonality, and regime alignment converge.

Final Thoughts

This exercise shows why visualization is important in finance: raw tables of returns and surprises wouldn’t reveal these patterns.

• Heatmaps instantly highlighted sector winners.

• Scatter plots demonstrated the weak surprise‑return connection. Bubble charts narrated the mega‑tech story over time.

• Violins unveiled the harsh truth  that markets are noisy. Cross‑sectional regime analysis reminded us that yesterday’s approach doesn’t ensure tomorrow’s returns.

The effort to interpret this data pays off: you shift from passive observation to active pattern recognition. You see not just what occurred, but where and when it happened. In trading and analysis, understanding the shape of complexity often surpasses having a perfect formula.

Visual storytelling turns data into intuition . And intuition, based on evidence, outperforms guesswork every time.

I have seen too many founders who are smart, legit, and ambitious get ghosted by investors simply because they couldn't walk through their unit economics.

It's not personal. It's math.

So here it is: Numbers that will either carry your pitch or quietly kill it, explained by someone who has sat through them time and time again, with examples, and no fluff.

Here’s what we’ll cover:

• 1. Burn Rate: How Fast Are You Lighting Your Cash on Fire?

• 2. Cash Runway: How Long Before You Run Out of Cash?

• 3. CAC (Customer Acquisition Cost): How Much Does it Cost to Convince Someone to Pay You?

• 4. Customer Lifetime Value (LTV): How Much is One Customer Worth Over Time?

• 5. Gross Profit Margin: What Do You Actually Keep After Delivering Your Service or Product?

• 6. Monthly / Annual Recurring Revenue (MRR / ARR)

• 7. Churn Rate: How Fast Are Your Users Leaving

• 8. Payback Period: How Long Before You Recover Your CAC?

• 9. Earnings Before Interest, Taxes, Depreciation, and Amortization (EBITDA)

• 10. Valuation: What’s Your Company Worth – and What Supports that Number?

• Real Talk Before You Close That Tab

  • Real Experience

  • Why This Matters

• Conclusion and Final Thoughts

1. Burn Rate: How Fast Are You Lighting Your Cash on Fire?

Burn rate is the speed at which a startup is spending its cash. Basically, how fast are you consuming your venture capital to cover over overhead until you generate positive cash flow from operations? It’s a measure of negative cash flow.

If you’re spending $80K a month to keep the lights on (payroll, AWS, your workspace snacks, and so on), that’s how much cash you’re burning each month. But many startups calculate two different burn rates: gross burn (how much cash you’re spending, ignoring any revenue), and net burn (monthly operating experiences minus any cash you take in each month). Net burn basically measures how fast your cash is shrinking, and it’s often what investors care more about.

Real talk: investors want to know when the plane runs out of fuel before they board. Thats what this metric helps them understand – how fast you’re going through the money you have.

At some point, if a company has a high burn rate, it has to reduce structural costs by cutting expenditures on labor, rent, marketing, and/or capital equipment. The burn rate is an important metric for any company, but it's particularly important for startups that aren't yet generating revenue. It tells managers and investors how fast the company is spending its capital

Watch this video to understand more about Burn Rate.

2. Cash Runway: How Long Before You Run Out of Cash?

Cash runway tells how long a startup can continue to operate at a certain burn rate until they are out of cash. For startups without revenue, you can calculate this by dividing available cash by total monthly expenses. Available cash is defined as the funds that are accessible now or can be accessed at a later time relatively quickly to pay for expenses.

When you’re making this calculation, it’s important to not include any anticipated fundraising and other uncertain sources of capital.

Actively managing cash runway is crucial for startup survival and growth. With a significant percentage of startups failing due to cash shortages, founders need to closely monitor their cash burn rate and runway.

The length of runway needed varies based on factors including the startup’s stage, industry, and milestones. In tighter venture capital markets, startups should plan for longer runways and consider strategies such as increasing revenue, reducing expenses, or raising additional capital. Regularly updating financial models and understanding metrics like the burn multiple can help you make informed decisions to extend your runway and align your growth ambitions with financial stability.

While it’s a simple calculation at face value, a cash runway analysis is nuanced and unique to every startup and can be impacted by a multitude of circumstances.

To calculate this, you just divide your total cash reserves by the amount you’re spending each month. Say you’ve got $250K in the bank and you’re spending $50K/month: 250/50 = 5. So you’ve got 5 months. Not 6. Not “it depends” – 5. That’s your runway.

Investors ask “If we don’t fund you, how long do you survive?” If you don’t know that answer, you're not fundraising – you’re freelancing with hope.

Here is a video that explains cash runway with real world examples and the thought process behind it.

And here’s an article from JP Morgan breaking down cash runway, its importance, and what can you to to maximize it.

Burn Rate vs Runway

So, let’s just make this super clear: burn rate is simply how much you spend each month to run your operation – that is, your negative cash flow. Runway is how many months there are left before your bank balance reaches zero.

So again, why do these numbers matter?

Because burn rate tells you how quickly you need to find more revenue or funding. Runway tells investors whether you are going to still be around by the time they finish their due diligence.

They are not just numbers. They are your survival clock.

Smart founders utilize these metrics to:

• Trim the fat without cutting muscle – know what to focus on and what to let go

• Forecast hiring/fundraising deadlines – know the process and prep for it. Numbers don’t line but they sure can get you ghosted.

• Assure investors you’re not going to come knocking again in 90 days – establishing credibility is key, make an investor realize its not just a hobby, you mean business.

The goal: Extend runway without stalling momentum. Keep the plane in the air, while building a bigger engine.

3. CAC (Customer Acquisition Cost): How Much Does it Cost to Convince Someone to Pay You?

Cost of acquisition refers to the entire cost that a business incurs to obtain a new client or asset. This includes the purchase price, shipping, installation, and marketing costs for the asset acquired. CAC takes into account the total expenditure on all marketing, advertising, and sales for the period, which you then divide by the number of new customers for the period.

In this case, all the upfront costs incurred to purchase a business asset, including equipment or inventory, are part of the cost of acquisition. Cost of acquisition includes:

• Purchase price of the item

• Costs to ship it to its point of use

• Costs to install the item

• Costs to get it up and running (in the case of equipment) or ready for sale (in the case of inventory) condition

• Marketing sales teams salaries

• All sales and consulting marketing expenses geared to get new consumers should all be included

Formula:
CAC = (Total Marketing + Sales Expenses) / Number of New Customers Acquired

Say you spent $10K last month across paid ads, content creation, outbound campaigns, and sales team costs. You onboarded 100 new customers, so your CAC = $100.

But is that good?

It depends on:

• Your pricing model (one-time vs. subscription)

• Your margin (how much of that sale do you actually keep?)

• Your customer retention (how long do they stick around?)

If you’re selling a $20 product once, a $100 CAC is a non-starter. But if that customer brings in $50/month for 12 months, you’ve got a solid return.

Watch for red flags:

• CAC is rising but revenue isn’t

• You’re overly reliant on paid ads (especially if organic/referral is flat)

• You don’t know CAC by channel (averages hide leaks)

A healthy CAC is one that pays itself back quickly and can be improved over time as you optimize funnels and messaging

Here is a video that breaks down CAC for you.

4. Customer Lifetime Value (LTV): How Much is One Customer Worth Over Time?

Customer Lifetime Value is the average monetary value of each customer to your business. LTV takes into account how much a unique customer is expected to spend with your business. It’s an important metric so you know how much new customers are worth to your business over their lifespan as a customer.

Let’s say you charge $25/month. The average customer sticks around 12 months.
LTV = $300.

In this case, if your CAC is $80? You’re in the green. But if it’s $350? You’re basically paying people to hang out (and losing money on them).

Now, let’s connect this to CAC.

Say your CAC is $80. You’re doing fine – your LTV is ~4x CAC. That’s what investors want to see.

Rule of thumb: you want your LTV to be at least 3x your CAC. A 1:1 ratio means you’re barely breaking even, before operational costs and the math stops working at scale. So if you can hit a 3:1 ratio, great – and based off my experience, your business will be much more appealing if it’s closer to 5:1.

And keep in mind that different models can have different thresholds. For example, a SaaS company with low churn can afford higher CACs, while an e-commerce platform might need faster payback. And marketplaces and freemium models may have lower LTV per user, but they can often more easily offset it with volume.

If you don’t know your LTV or can’t defend it with data, it becomes hard to justify spend – and easy for investors to walk.

If you want to know more, this video walks you through the basics.

And here’s a video that beautifully explains the CAC and LTV relationship.

5. Gross Profit Margin: What Do You Actually Keep After Delivering Your Service or Product?

Gross profit margin shows the amount of money a business collects after it pays for all its expenses. It’s usually calculated as a percentage of sales. This specific metric is also referred to as the gross margin ratio.

Companies use gross margin as a measure of how production costs relate to revenue. If a company's gross margin falls because it is making less revenue, it may try to cut labor costs, find cheaper suppliers of materials, or increase prices to increase revenue.

Gross profit margins can also allow a business to measure how efficient a company is, or compare two very differently sized companies that share a common revenue stream or product

If you sell a subscription for $50/month and it costs you $10/month to host, maintain, and support it, your gross margin is 80%.

• Good: SaaS companies often hit 70–90%.

• Bad: If you're below 30%, your "scalable" business will collapse under weight.

Want to know the conceptual math behind this metric and how it differs from Profit Margin? Here is a fantastic video that easily breaks it down.

6. Monthly / Annual Recurring Revenue (MRR / ARR)

Annual recurring revenue (ARR) is revenue a company expects to see from its product and service offerings, calculated over the course of a year. Companies that sell annual subscriptions like using ARR as a sales metric to track what they anticipate making in a year.

ARR tends to be used if companies sell a product or service in the software as a service (SaaS) space, but it can also be useful in terms of streaming services, cell phone bills, and (almost) anything else with a predictable, recurring charge.

ARR is calculated annually, whereas monthly recurring revenue (MRR) is calculated monthly. MRR is useful in that it shows what’s happening on a month-to-month basis. For example, if you change your price in April, you can see the immediate effects of that change in May. MRR also helps track fluctuations in revenue based on outside factors like holiday shopping seasons and economic conditions.

In a nutshell, Monthly / Annual Recurring Revenue = predictable income.

If you’re pulling $20K/month in subscriptions, that’s $240K ARR. Simple.

What investors care about:

• • Is it growing?

    • How fast?

    • And how stable is it?

Here is a founder breaking down the metric and explaining the relationship between MRR/ARR.

7. Churn Rate: How Fast Are Your Users Leaving

The churn rate, also known as attrition rate, represents the rate at which a customer stops doing business with a company. Customer churn is typically expressed as the percentage of service subscribers that discontinue their service subscriptions within a time frame. Churn can also be expressed as the rate at which employees leave their jobs in a given time.

In order for a business to grow its number of clients, its growth rate (which takes into account new customers) must be higher than its churn rate.

The benefit of calculating a churn rate is that it can clarify how well a business is retaining its customers, which is a measure of the quality of service the business is providing and the usefulness of that service.

When a business can see its churn rate increasing from period to period, this suggests that a critical aspect of how it is running the business might be problematic or flawed.

It could be the result of:

• A faulty product(s)

• Bad customer service

• Costs exceed utility to customers

And so on.

The churn rate will indicate to a business that it needs to learn why its customers are leaving, and where it needs to adjust its business. It’s more expensive to attract new customers than it is to retain them, so reducing the churn rate can save a business resources in the future.

Real talk: Say you had 500 users at the start of the month, and you lost 50 by the end of the month. That’s 10% churn – which is high! Annualize that and…ouch. You're not growing. You're replacing.

Make sure you fix this before you fundraise. Or at least explain why churn’s high and what you’re doing to plug the holes.

Here is a video that beautifully explains Churn Rate.

8. Payback Period: How Long Before You Recover Your CAC?

The payback period is a popular tool for determining investment return. People invest money for the purpose of getting it back and generating a positive return on the money they invested. The shorter the payback period, the more beneficial the investment will be.

The payback period does not factor in the time value of money. You can determine it simply by counting the number of years until the principal paid in is returned.

This metric measures how quickly your customer pays you back for the cost of acquiring them. The payback period doesn’t take into account the total profitability of an investment. It’s just concerned with paying the investment back.

There are two common interpretations:

1. Customer-Level Payback: If your CAC is $250 and your customer pays $50/month, it’ll take 5 months to recover the acquisition cost.

2. Investment-Level Payback: You spend $100,000 on a new sales hire, tool stack, or feature. You want to know how long it takes for that investment to generate $100,000 in profit.

Both use the same principle: the shorter the payback period, the less cash you need to float your growth.

If you want a target, aim for 6 months for customer-level payback. Closer to 3-6 is ideal. Long payback periods mean you need deep pockets – or exceptional retention – to stay afloat.

Here’s a video where you can learn more.

9. Earnings Before Interest, Taxes, Depreciation, and Amortization (EBITDA)

EBITDA stands for Earning Before Interest, Taxes, Depreciation, and Amortization. You can think of this as just your company's operating profit if you wanted a very rudimentary way of referring to it.

It’s not flashy. It’s not fun. But it tells investors: “Here’s what we really make once the accounting fog clears.” and “Are we generating real profits from our actual operations?”

EBITDA is what investors look at because it is the best way of comparing apples to apples when considering startups. EBITDA can provide investors a measure of your operational health.

A negative EBITDA for an early stage company isn't going to raise any eyebrows. Just don’t act surprised when someone brings it up. You need to do that math before the pitch. But, if you have a growing early stage company that's moving from negative EBITDA to positive? Now your getting into grown folks business.

Here’s a video that explains the basics of EBITDA.

And here’s another video that explains how investors look at EBITDA and its value in determining your business’s worth.

10. Valuation: What’s Your Company Worth – and What Supports that Number?

Valuation is a focused exercise that determines the value of an asset, investment, or company. So, how much your company’s worth. And the goal is typically to determine whether that value is a fair value.

Valuations can be conducted in one of two ways:

• an absolute valuation, which evaluates a company on its own merits and entirely independently of other factors/companies, or

• a relative valuation, which evaluates the company relative to other similar firms, or assets, in the same sector or industry. This determines if the company, or asset, is worth that much relative to others.

Depending on how the analysis and conclusions are reached, there are a variety of methods and techniques used to develop valuations. And as you’d expect, there’s often significant variability between outputs (or valuations) based on the inputs and context.

While valuations are predominantly quantitatively driven, there’s often a significant subjective influence that come from the assumptions and estimates made along the way. Valuations are also subject to developing situations and events outside of the analysis or the control of the analyst – for example, earnings reports or material news, or economic news – that can result in a change to a valuation stance.

If you’re pre-revenue and you’re saying $30M because a friend raised at that, please stop. Their experience likely has nothing to do with yours.

Valuation = traction + market comps + revenue + momentum + team.

Valuation isn’t just about what you want – it’s about what you can defend.

Startups are typically valued using:

• Comparable Analysis (Comps): What similar companies are worth

• Discounted Cash Flow (DCF): Projecting future cash and discounting it back

• Revenue Multiples: Often 5x–10x for SaaS, but varies wildly

• Precedent Transactions: What investors paid in past rounds for similar startups

But that’s the math.

Here’s the messy truth: Valuation = Traction + Team + TAM (total addressable market) + Timing + Storytelling.

Hard factors:

• MRR/ARR

• Growth rate

• Churn

• CAC:LTV

• Gross margins

Soft factors:

• Founding team’s track record

• Market momentum

• Hype or scarcity

Don’t inflate. Don’t anchor to your friend’s raise. Know your comps. And show why your model is defensible, not just desirable. Inflated numbers make investors run. They don’t correct you – they just ghost you.

There are numerous books written on valuation and each technique could be its own PhD. But my role here is to give you a sneak peak into the metrics.

Here’s a basic video on valuation if you’re interested in a deeper dive.

And here’s a more detailed video course outlining different forms of valuation. Professor Damodaran from New York University is considered to be one of the aces and thought leaders when it comes to valuation. In this video course he explains stepwise and beautifully so you can understand and explore the fascinating world of valuations.

Real Talk Before You Close That Tab

Real Experience

I met a founder once – early days, rough product, but you could tell he actually cared. He wasn't trying to look good. No buzzwords. No "disrupt" talk. Just someone trying to solve something annoying and important.

He walked into the room with a twinkle. Not swagger – just that gentle intensity. We were leaning in.

Then, in the middle of the pitch, someone asked, "So what's your monthly burn?" And I swear to you, he said, "Umm... I think my co-founder has that. I haven't looked in a while."

That was it.

No freak out. No awkward pause. Just... a cluck. Like a window closing in the background.

The product? Still smart. But the moment? Gone.

Nobody was mad. Nobody laughed. We even said thank you. But nobody followed up.

Why? Because it didn't feel like a business. It felt like a maybe.

Why This Matters

I’ve seen so many versions of that same scene play out. It’s never about charisma. It’s not even about the idea, half the time.

It’s about whether the person asking for money actually knows what they’re building. Not the dream, the mechanics. The guts, nuts and bolds of the business. The ugly Excel math nobody brags about on Twitter.

Unfortunately, no simple pitch deck will do that part for you. No co-founder can answer those questions on your behalf.

If it’s your vision, own the math. If it’s your company, learn the cost of keeping it alive.

The rest? The logos, the taglines, the “go-to-market” plans?.... All of that’s just packaging.

And you don’t have to be perfect either. You just have to be in it. Eyes open. Numbers in your head.
Because if you’re asking people to believe in what you’re building, you’d better believe in the scaffolding holding it up.

So yeah, know your CAC. Your LTV. Your margins. Your churn. Not to check some box on an investor’s sheet, but to prove to yourself and the investor that the thing you’re spending your life on…has legs. That it can stand. And run.

And maybe, someday, outlast you. Maybe!

Conclusion and Final Thoughts

I hope this was helpful to you, especially if you’re a founder or aspiring founder trying to build the next big thing. While there a many more ratios and concepts, these are the crux of them.

A lot of other complex ratios and valuations are either built using these metrics or refer them in some way. And each of these metrics could be an article of its own. But I wanted to give you my top 10 run down so that you could get a head start. Numbers are very much a part of the ideation stage itself, and omitting them from your strategy could prove to be a fatal mistake.

I’ll leave you with one last video on How to Start a Start Up with Michael Seibel (Reddit, YC, Twitch) that I hope you find valuable. It lays out, in a crash course format, the mindset of a founder who has been there and done that. The fun fact is that a lot of the themes he speaks of tie in to the metrics here, directly or indirectly.

I hope this gives you a perspective of being on the other side, evaluating your hard work and passion, and I hope it sets you up for success in your next Investor Review.

I look forward to your thoughts, comments, and feedback. If this was helpful, engaging, and informative, do share it – you never know who may need it, or could benefit from it. I wish you all the very best in your funding rounds.

Until then, keep learning, unlearning, and relearning, folks.

It's particularly important in finance, especially in these areas:

• Securitized debt
• Exchange-traded options
• Credit default swaps
• Over-the-counter derivatives securities

In this article, you'll learn why the Black-Scholes Equation is so important in finance, what problems it solves, and the industries it created.

Here's what we'll cover:

• Prerequisite knowledge of finance
• Analogy: Predict the price of a ticket for a concert
• Plain English explanation with code example
• Implications in the real world

Note: In the code example, we will be working with European call and put options.

Prerequisite Knowledge of Finance

To get the most out of this article and understand the Black-Scholes Equation, you just need to know what financial derivatives and options are in finance.

Essentially, financial derivatives are tools investors use to manage risks and improve returns.

There are many types of financial derivatives. One of these is called options.

Options are like financial choices. With options, you can get the right to buy or sell something at a certain time and price, but only if you want to.

The main idea is that they help manage risk so you can make future better investments.

Analogy: Predict the Price of a Ticket for a Concert

Imagine you are planning to buy a ticket for a concert.

The ticket prices change depending on the popularity of the artist, demand, and time until the concert.

Depending on that, you will make the best possible decision to buy the ticket at the lowest price.

Just as you think about the risk of buying the thicket at a certain time, investors use the Black-Scholes Equation to estimate the fair value of financial derivatives.

This way, they make sure they make wise investment choices in ever changing markets.

Black-Scholes in Plain English – with a Code Example

Essentially, the Black-Scholes Equation solved the problem of how to price options correctly in financial markets.

This is very important, because it helps banks and financial institutions effectively manage risk.

However, it was not always like this. Before 1973, when the equation was created (its creators won a Nobel prize), determining the price of options was much more complicated and difficult.

Before the creation of the Black–Scholes equation, there wasn't a standardized mathematical method to predict the prices of options.

Traders often relied on personal experience and market conditions, which led to unreliable option prices.

And earlier mathematical methods did not fully consider factors like volatility, time decay, and interest rates. So there was a lot of error when pricing options.

Here is the Black-Scholes Equation:

$$\frac{\partial V}{\partial t} + \frac{1}{2}\sigma^2 S^2 \frac{\partial^2 V}{\partial S^2} = rV - r S \frac{\partial V}{\partial S}$$

While we won't look very deeply at the equation itself, we will outline its key components and implications.

Essentially, the Black-Scholes equation predicts how an option's value changes over time based on several variables:

• V - Price of option as a function of stock price S and time t
• S – Price of the underlying asset
• t – Time
• σ – Volatility
• r – Interest rate.

The left side of the equation explains how the option's value changes over time and how market ups and downs affect it.

The right side of the equation shows how the option's value increases due to interest rates and how changes in the asset's price impact it.

By making these two sides equal, we figure out the fair price of the option.

Python Code Example

In this code example, we will find, based on many parameters, the theoretical market value of an option.

For our example, let's assume the following:

• Current stock price (S) = $100. This is the price of the stock right now.
• Strike price (K) = $105. This is the specific price at which the option holder can buy (call) or sell (put) the underlying asset.
• Time to expiration (T) = 1 year (or 1.0 when expressed in years). This is the time left until the option expires.
• Risk-free interest rate (r) = 0.05% (or 0.0005 when expressed as a decimal). This is the interest rate on a risk-free investment.
• Volatility (sigma) = 20% (or 0.2 when expressed as a decimal). This is how much the stock price is expected to fluctuate.

from blackscholes import BlackScholesCall, BlackScholesPut

def calculate_option_prices(S, K, T, r, sigma, q):
    """
    Calculate the Black-Scholes option prices for European call and put options using the 'blackscholes' package.

    Parameters:
    S : float - current stock price
    K : float - strike price of the option
    T : float - time to maturity (in years)
    r : float - risk-free interest rate (annual as a decimal)
    sigma : float - volatility of the underlying stock (annual as a decimal)
    q : float - annual dividend yield (as a decimal)

    Returns:
    tuple - (call price, put price)
    """
    # Creating instances of BlackScholesCall and BlackScholesPut
    call_option = BlackScholesCall(S=S, K=K, T=T, r=r, sigma=sigma, q=q)
    put_option = BlackScholesPut(S=S, K=K, T=T, r=r, sigma=sigma, q=q)

    # Get call and put prices
    call_price = call_option.price()
    put_price = put_option.price()

    return call_price, put_price


call_price, put_price = calculate_option_prices(100, 105, 1, 0.0005, 0.20, 0.0)
print("Call Price: {:.6f}, Put Price: {:.6f}".format(call_price, put_price))

Now let's examine the code more closely and see what's really going on here:

Step 1: Import the Library

This is the Python library we are using in this article:

from blackscholes import BlackScholesCall, BlackScholesPut

Importing functions

Step 2: Create the Function to Calculate Options Prices

In the below code, we are importing the function we need to calculate the options call and put prices.

def calculate_option_prices(S, K, T, r, sigma, q):

    call_option = BlackScholesCall(S=S, K=K, T=T, r=r, sigma=sigma, q=q)
    put_option = BlackScholesPut(S=S, K=K, T=T, r=r, sigma=sigma, q=q)

    call_price = call_option.price()
    put_price = put_option.price()

    return call_price, put_price

Function to calculate call and put prices

The main parameters of the function are:

• S : float – current stock price
• K : float – strike price of the option
• T : float – time to maturity (in years)
• r : float – risk-free interest rate (annual as a decimal)
• sigma : float – volatility of the underlying stock (annual as a decimal)
• q : float – annual dividend yield (as a decimal)

And it returns:

• tuple – (call price, put price)

First, we calculate the call and put options. Then we extract the price from it. We can also get other characteristics like the charm or the delta of these financial contracts according to the library documentation.

Step 3: Calculate the Options Pricing

The call and put prices of an option are the costs to buy the respective option contracts.

call_price, put_price = calculate_option_prices(100, 105, 1, 0.0005, 0.20, 0.0)
print("Call Price: {:.6f}, Put Price: {:.6f}".format(call_price, put_price))

Applying the function

We use as examples:

• Current stock price: $100
• Strike price: $105
• Time to maturity: 1 year
• Risk-free interest rate: 0.05% (as a decimal: 0.0005)
• Volatility: 20% (as a decimal: 0.20)
• Dividend yield: 0%

Based on this factors, we price:

• Call Option Price: 5.924799
• Put Option Price: 10.872312

Which means that, given these parameters:

• The price at which you have the right, but not the obligation, to buy is 5.924799 dollars
• The price at which you have the right, but not the obligation, to sell is 10.872312 dollars

Implications in the Real World

The equation has had a massive impact in the world of finance.

Below are some of the industries the Black-Scholes Equation has changed greatly:

Securitized Debt

In simple terms, securitized debt refers to turning loans into something that can be bought and sold.

The Black-Scholes equation changed the way banks price grouped-up debt, like mortgages.

Before the Black-Scholes equation, it was very hard to know the worth of these debts. But with the equation, banks can understand their value much better. This made it easier to buy and sell these debts while knowing the potential benefits and risks.

This way, the market for these mortgage debts grew. Which in turn helped grow the housing market.

Exchange Traded Options

Trading options was a very uncertain business. There was no way of truly knowing how to correctly price them.

However, with the Black-Scholes equation, option pricing became far easier. It allowed people to calculate an option based on an underlying asset's price, volatility, time to expiration, and interest rates.

The newfound precision helped grow the options market.

Credit Default Swaps

Credit default swaps are like insurance policies for loans. With a credit default swap, you are protected if the borrower fails to pay back.

Credit default swaps are very important in managing credit risk. But it was only after the black Scholes equation was created that they were accurately priced.

This way, credit default swaps became a very important tool for financial institutions for financial risk management.

Over the Counter Derivatives Securities

Over-the-counter (OTC) derivatives are private deals made between two parties without a stock exchange.

Before Black-Scholes, negotiating the terms and prices of OTC derivatives was very hard. But then the Black-Scholes equation offered a standard way of finding the price of derivatives.

This allowed market participants to negotiate contracts more efficiently and accurately.

Conclusion

The Black-Scholes equation helped create more precision in the way certain things are priced.

This precision helped create more stable institutions, which in turn helped create a more resilient economy.

If interested in learning more, see this video:

If you are interested in learning more about finance:

Full code

Malindi grew up on a farm in rural Kansas, in the middle of the US. She trained to become a diplomat, and volunteered overseas. But along the way, she discovered a love of math and computer science. That passion has landed her jobs in New York City, London, and San Francisco.

I met up with Malindi in downtown Manhattan to learn all about investment banking, and how she modernized her department at JP Morgan using her software engineering skills. We talk about the high-stakes world of global finance, where she was executing trades sometimes worth hundreds of millions of dollars.

We also talk about her time as a venture capitalist. She researched thousands of startups to decide which ones her fund should invest in.

This is one of the most technical interviews I've done. I've done my best to make Malindi's world of math, AI, and high finance as accessible as I can. I hope you enjoy it.

Malindi on LinkedIn: https://www.linkedin.com/in/alexandra-malindi-colyer-46b95589/

This article will explain:

1. Who should use a ROM
2. When to use a ROM
3. How to calculate a ROM
4. Simple example of a ROM

Who Should Use a ROM

Project Managers use rough orders of magnitude to determine initial cost ranges for upcoming projects. Many organizations and most software organizations use project managers to lead projects.

Their role involves leading their project team, defining the goals of the project, reporting on the progress to the company stakeholders, and seeing the project through to completion.

man with blank stare listening to man trying to explain something

Who else can use a ROM? You! 👊 As you'll see, the concept of a ROM is very simple to grasp and can be useful in myriad circumstances.

When to Use a ROM

ROMs are extremely important for project managers in the early stages of a project's planning. They're used to decide whether or not a project is going to be feasible or not.

Rough Orders of Magnitude can be used to prepare a project business case (an analysis of the potential benefits of the project) before a project is committed to.

How to Calculate a ROM

Calculating a ROM is pretty straightforward. It is meant to provide a range of the possible cost of a project and you can find it using the following formulas:

Lower Bound = ROM Estimate x 0.75

Upper Bound = ROM Estimate x 1.75

Fred Flinstone counting on fingers

During the planning phases of a project, the project manager can come up with an estimate of the costs involved. From here, using the above formulas, they can then calculate a lower boundary in case the project ends up being less expensive and an upper boundary in case it goes over budget.

Since it is helpful to have more room on the upper boundary to avoid running out of funding if it does go over budget, the rough order of magnitude's upper boundary is considerably higher.

ROM Example

You've probably calculated a ROM without knowing it. 💡

Say you're going to the movies with a friend. You figure there will be ticket costs plus refreshment costs. Maybe $20 a ticket since you're going to the latest Nolan IMAX film and then another $30 a person since popcorn is more expensive than steaks at the movies.

This puts you at $100 total for both of you.

The Lower Bound in a ROM calculation would be $100 x 0.75 = $75.

The Upper Bound would be $100 x 1.75 = $175.

You probably don't do this exact math in your head, but you likely have figured there could be extra expenses. You go into the date thinking you could spend as much as $150, for instance.

Angry Bird eating popcorn at movies

Conclusion

Rough Orders of Magnitude are extremely important in organizational projects. It is vital to decide at the onset whether or not the project is feasible. And the principals behind ROMs are both common sense and easy to calculate.

I hope this has been helpful to you!

You can find and follow me on LinkedIn. I'd love if you said hey! 👋

CapEx stands for capital expenditures, which is money used by an organization to purchase, improve, or sustain physical assets.

Commonly held physical assets include PP&E (property, plans, buildings, and equipment for running and sustaining the business).

In order to explain CapEx further I want to use a fictitious business to illustrate examples in a clear and concise way.

What is CapEx?

Firstly, imagine a company that makes jewelry - like engagement rings. In order to produce rings, the company needs basic materials (gold, diamonds, etc), equipment to turn these raw ingredients into final products, and labor.

Now imagine you own and operate this business.

As the owner of this company you decide that you will buy property (perhaps a warehouse for storing raw materials), equipment (such as a high temperature furnace to smelt gold), or fans to keep the showroom cool and comfortable.

What would you call this property and equipment? How would you account for them? To put it simply, this equipment is a capital expenditure.

Why?

In this case you are quite literally expending capital (money) in exchange for other types of assets to help the business grow.

And that is precisely what CapEx is.

Because buying the machinery, equipment, and property would help the business maintain or increase its operation, we classify these transactions as CapEx.

Another example is Goldspot Pens, a fountain pen store that sells bottled ink and fountain pens, who are investing in new, bigger warehouses for storing their fountain pens and ink. This costs more money, but increases Goldspot Pens' scope of economic performance in the future.

In this simple example, the choice to buy these future economically productive assets represents CapEx.

In summary, CapEx is the money an organization spends to buy, maintain, or improve its assets to increase its scope and economic performance.

How to Calculate CapEx

Now that you know what CapEx is, and are armed with an example of CapEx at a jewelry business, you might be curious how a company calculates CapEx in practice.

There is a simple equation: CapEx is equal to changes in PP&E added to Current Depreciation.

The formula: CapEx = ΔPP&E + Current Depreciation

Let’s break down these parts of the equation to clarify.

1. CapEx: spending on business critical property or equipment or technology to grow the firm.
2. ΔPP&E: changes in PP&E recorded on the balance sheet. If a firm buys a machine for $1M, for example, it will mark this machine in the PP&E section (usually under long term assets).
3. Current Depreciation: a concept that captures the cost of the asset by the useful life of the asset.

Specifically, here are the steps to calculate Current Depreciation.

• Declare the useful life of the asset (10 years is a fair value to use).
• Subtract the salvage value of the asset (that is, at the end of year 10 the machine will be worth some value).
• Divide this value by the starting cost of the asset ($1,000,000)

In this example, let me show you what a depreciation table will yield. The asset starts at $1,000,000 on Day 1 but ends the depreciation cycle at $25,000.

Now that you know how to calculate depreciation, you can solve CapEx mathematically, using either a Direct or Indirect Method.

Direct method:

• Amount spent on asset #1
• Plus: Amount spent on asset #2
• Plus: Amount spent on asset #3
• Less: Value received for assets that were sold
• = Net CapEx

Indirect Method:

• PP&E Balance in the current period
• Less: PP&E balance in the previous period
• Plus: Depreciation in the current period
• = Net CapEx

If you bought an asset for $1,000,000 and sold it at the end of year 10 for $25,000 your Capital Expenditure would be: $1,000,000 - $25,000 = $975,000.

The Meaning of CapEx in Practice

When you decide to buy equipment today, you are doing so because you believe that the equipment will generate future economic benefits, or profits.

If you didn’t believe this, you likely would not have purchased the property, hardware, or equipment in the first place. Because CapEx is any type of expense that a company capitalizes, or shows on its balance sheet as an investment, you will need to justify how the purchase adds economic value to the firm's future.

For this reason, CapEx is considered a capitalized expenditure and not an expense.

This enables you, as the business owner, to match the economic benefits of the items you are buying with the costs in a given period of time.

If you were unable to match the purchase with economic benefits, you would treat the expenditure as an expense and not as an investment.

Summary

Many businesses, but not all, have CapEx.

Think about large companies like Google, AT&T, and Apple with large data centers, 5G networks, and fulfillment centers, respectively. These are all examples of CapEx.

These companies invest in these pieces of land, equipment, and machinery because each firm wants to secure a profitable future.

And without this CapEx, such a future would be harder to obtain.

Let’s say you want to buy a car. But you don’t have enough money sitting around to pay for the car in one payment.

So you take out an auto-loan. This loan provides you borrowed money with the explicit designation that it be used to purchase the vehicle.

Here is a second example. Imagine you want to buy a wedding ring but don’t have enough funds to pay for the ring up front today. What do you do? You get a small loan.

Sounds simple, right?

A mortgage is the exact same thing, except applied to a property (a home or building). In simple terms, a mortgage is a type of loan, just like an auto-loan or financing for jewelry.

Specifically it is a loan in which a person borrows money to buy or refinance a house. That’s it.

A loan can be used to describe many different types of financial transactions. You can have a student loan, or per the example above, a loan to buy a car.

If you are borrowing money to buy a home, you will need a home loan. And home loans are called mortgages.

Who needs a mortgage?

Most people buying property require a mortgage to complete the transaction. That is because most people don’t have enough money available to spend hundreds of thousands or millions of dollars at one time.

If you don’t have the cash upfront, a mortgage is the loan type you will leverage to complete the transaction.

Because a mortgage is usually paid back over many years (30 is typical) it enables buyers to make a purchase using borrowed money.

As the years go on, the mortgage is repaid. If you are unable to make your mortgage payments, the lender can take the home in a process called foreclosure.

This is because mortgages are “secured” by the asset being purchased (that is, a home).

How is a mortgage structured?

There are different types of mortgages. But to use simple math, let’s assume that an average American home costs $300,000.

If you want to put down 20%, you will need $60,000 in cash. This $60,000 represents the down payment. The size of the down payment will depend on many variables - your credit score, the type of loan you are applying for, and the value of the property itself.

The rest of the purchase price - or $240,000 - will be financed by the mortgage. When borrowing this $240,000, you will need to agree to different terms of repayment.

Common terms will include the time that the loan must be paid back, the interest rate of the loan, and what happens if you fail to make loan payments on time.

The common parties involved in this transaction include:

1. The lender: this is the bank or financial institution that is providing you money.
2. The borrower: the person getting the loan.
3. Loan servicer: the organization in charge of providing monthly mortgage statements and processing payments, and so on.

How to pay back your mortgage

As you are thinking about a mortgage - and if you should apply for one (and the size) - you will want to think about how much the loan will cost you. When borrowing money you need to be prepared to pay it back with interest.

One way to think about this interest is using a framework called the “cost of capital”.

In other words, how much is it costing you to borrow this money? The cost of capital is the cost of debt if your home is financed solely through debt.

When paying back a mortgage you will have four costs:

1. Principal (paying back the principal of what was borrowed)
2. Interest (paying the interest on the loan itself)
3. Taxes
4. Insurance (most common for people who put down less than 20% of the home value)

Some mortgages use fixed rates and others adjustable rates. Fixed rates are conceptually easiest to understand: you will pay a fixed (or constant) rate until the mortgage is paid back in full.

Using the $240,000 example from above and an interest rate of 3.5% over 30 years, you will need to pay $1,078 a month, every month, to pay the loan back.

Variable interest rates are slightly more complex. The interest rate is not fixed but rather benchmarked to a reference rate.

Sometimes people start their mortgage payment with a fixed interest rate followed by a variable one.

How the mortgage process works

Conceptually, there are three steps to getting a mortgage. This article is not the definitive guide, by any means, and there are complex nuances I am overlooking.

Step 1: Get approved for a mortgage. Sometimes this requires pre-approval. In short, its an agreement by a bank to loan you money,

Step 2: Find a home you want to buy, make an offer, and figure out how much money you will need to borrow to purchase this property.

Step 3: Get final approval and close on the loan.

Bringing It All Together: A Mortgage and Its Value

A loan is when money is given to you in exchange for repayment of the loan amount plus interest. One type of loan is a mortgage.

If you ever get confused and need a simpler example, just think about the example of buying a ring: you borrow money today to buy something you want or need.

And then you pay back that loan over time.

How much you borrow and on what terms (interest rate, length of repayment time table, and so on) are important considerations to understand before moving forward with this type of financial instrument.

What is a 401K? Here is the legal and technical answer: in the United States, a 401k plan is an employer-sponsored contribution pension account defined in subsection 401k of the Internal Revenue Code.

A bit confused?

Let me break it down further in plain English.

A 401k is a pension account. A pension is any plan or fund which provides retirement income. So a 401k is dedicated to helping you save for retirement.

Now that you know what a 401k is, you might still be wondering why you should use it and how to set one up.

I have answers to both questions.

Why should you have a 401k?

A 401k is used to defer income today to save for tomorrow. In order to set up this type of account, you need to be employed and your employer needs to have a 401k plan that it offers to its employees.

The U.S. government enables employees to make tax-deferred contributions to their retirement account every year. As of calendar year 2021, the annual contribution limit for an employee under the age of 50 is $19,500.

So this means that, if you're under 50 years old, the most you can contribute to your 401K plan is $19,500 a year.

You might be wondering why the IRS limits annual contributions. There are two reasons:

1. The IRS wants to prevent highly paid workers from benefitting more than the average worker from the tax advantages they provide.
2. The government wants people to spend some of their income today.

IRA Contribution Limit for 2020 (and every year back to 2002)

Here is a table showing historical limits. As you can see, each year the total contribution limit goes up slightly to account for inflation and increases wages.

A note about this table: as an employee, you are not able to contribute more than $19,500 of your annual salary to your 401k if you are under 50 years old. If you are older than 50, however, the government enables you to make larger savings contributions.

This policy is intended to accelerate savings from people closer to retirement age.

Employers could contribute up to $57,000 (catch-up at $63,500) in 2020; in 2021 that amount has risen to $58,000 (or $64,500 with the catch-up contribution).

Why you should contribute to a 401k

If you are researching 401k contribution limits, perhaps you already understand the benefits of this savings account. If not, let me outline three reasons why a 401k can help you plan for retirement.

You can leverage these points if you are looking for finance feedback or if you're becoming better informed about how to plan for retirement.

1. It is easy and painless. Money can be deducted from your paycheck in regular intervals. No need to engage in manual saving.
2. Many firms offer some sort of employer match as a financial incentive to save. Matching is a perk, like leadership development or office food. It is provided at the discretion of the firm and can be retracted. So if you take advantage of matching you will obtain even greater funds.
3. Tax breaks and benefits: your paycheck contributions are tax deductible. Because the capital you contribute doesn’t count towards your gross income, your net tax bill is lowered. Secondly, the money that is saved is tax-deferred. Over the course of your career these deferred taxes will add up.

Summary

In this article we defined:

1. What a 401k is
2. How these pension accounts work
3. The annual contribution limits of a 401k
4. And the benefits of these accounts for your saving aspirations.

If you want to plan for your financial future, a 401k helps you make forced savings. This way you don't end up spending all of your income on travels or nights out.

By taking a disciplined approach and maximizing savings over time, a 401k can help you plan for the future you want by preserving and protecting future cash flows.

We've released a complete course on the freeCodeCamp.org YouTube channel that will teach you the basics of algorithmic trading.

Nick McCullum developed this course. Nick has worked as an investment analyst, portfolio manager, and software developer at financial startups for his entire career. He has a knack for explaining complex investment topics in a way that beginners can understand.

This course uses Python. Python is the most popular programming language for algorithmic trading. Python is powerful but relatively slow, so the Python often triggers code that runs in other languages.

Along with Python, this course uses the NumPy library to speed up the code. NumPy is the most popular Python library for performing numerical computing. Although NumPy is written for use in Python, the core underlying functionality is written in C, which is a much faster language.

Note that this course is meant for educational purposes only. The data and information presented in this video is not investment advice. One benefit of this course is that you get access to unlimited scrambled test data (rather than live production data), so that you can experiment as much as you want without risking any money or paying any fees.

Course Outline

In this course you will first learn the basics of algorithmic trading. Then you will learn how the IEX Cloud API works. We will use the API to gather data.

The bulk of this course teaches how to build three algorithmic trading projects.

Algorithmic Trading Fundamentals and API Basics

• What is Algorithmic Trading?
• The Differences Between Real-World Algorithmic Trading and This Course
• The Basics of API Requests

Building An Equal-Weight S&P 500 Index Fund

• Theory & Concepts
• Importing our Constituents
• Pulling Data For Our Constituents
• Calculating Weights
• Generating Our Output File
• Additional Project Ideas

Building A Quantitative Momentum Investing Strategy

• Theory & Concepts
• Pulling Data For Our Constituents
• Calculating Weights
• Generating Our Output File
• Additional Project Ideas

Building A Quantitative Value Investing Strategy

• Theory & Concepts
• Importing our Constituents
• Pulling Data For Our Constituents
• Calculating Weights
• Generating Our Output File
• Additional Project Ideas

The first project in the course is an equal-weight S&P 500 screener. The S&P 500 is the world's most popular stock market index. In this project, you will build an alternative version of the S&P 500 Index Fund where each company has the same weighting.

The second project is a quantitative momentum screener. Momentum investing means investing in assets that have increased in price the most. You will create an algorithm that implements this strategy. First, you will build a strategy that uses a single momentum metric. Then, you will expand to build a more sophisticated strategy that uses multiple metrics together.

The final project is a quantitative value screener. Value investing means investing in stocks that are trading below their perceived intrinsic value. Like the previous project, you will first build a strategy that uses 1 value metric. Then, you will expand to build a more sophisticated strategy that uses 5 different value metrics together.

How to Get Started with the Course

This course is original content created by our nonprofit, freeCodeCamp.org. It was made possible a grant provided by IEX Cloud, and with market data they provided us. Any opinions or assertions contained herein do not represent the opinions or beliefs of IEX Cloud, its third-party data providers, or any of its affiliates or employees.

📺 You can watch the full course on the freeCodeCamp.org YouTube channel (4-hour watch).

And you can access the full open source course files, with both starter files and finished files, at this GitHub repository. Happy coding.

While the use of NFC for things like contactless payments was growing steadily, it exploded early this year due to the Coronavirus pandemic.

In this article we'll go over what NFC is, what it's used for, some creative ways to use NFC, and more.

What is NFC and how does it work?

NFC stands for near-field communication. It is a standard for devices to communicate with each other wirelessly from a very close distance.

NFC is a subset of another technology called RFID, so let's dig a bit into that before circling back to NFC.

What is RFID?

Radio-frequency identification, or RFID, is a generic term for technologies that use radio waves from a reader to track specific tags. These tags all include an antenna and a tiny chip, and can come in many shapes and sizes.

Highway toll payment devices and those plastic things on clothes and other expensive items in stores are some common examples of RFID tags.

A diagram of an RFID tag. NFC tags look very similar – Source

If you've ever seen those big devices on either side of a store entrance, those are just big RFID readers. They're constantly transmitting radio waves and listening for a response.

So what happens if you try to leave a store and there's still a tag on the item you bought?

Most RFID tags are unpowered, so when the antenna in the tag picks up radio waves from the reader, it generates a small amount of electricity. That electricity activates the chip inside the tag, and it sends a signal with the information stored on the chip back to the reader.

In this case, when the reader receives a signal back from the tag on your item, it sounds an alarm.

How are RFID and NFC related?

NFC is a newer, high-frequency version of RFID, and also involves both tags and readers.

NFC's higher frequency means that, while it can transfer data much faster than RFID, it only works from a distance of about 4 cm/1.6 in or less. Meanwhile, RFID works from a distance of up to 12 m/40 ft.

What is NFC used for?

There are a lot of use cases for NFC, but here are some of the most common you'll see.

Contactless payments

These days, the most common thing that NFC is used for is contactless payment. Many newer credit and debit cards include an NFC tag, so you can just hold your card just above a payment terminal rather than swipe or insert it.

Contactless payment enabled credit and debit cards have a symbol on them similar to these:

_Most contactless payment cards will have a similar symbol on the front or back – Source_

Most modern phones include an NFC chip, which can act as both an NFC reader/writer and tag.

This chip, paired with a mobile payment app like Google Pay, Apple Pay, and Samsung Pay, means that you might not even need to take your wallet out anymore.

Instead, your phone can act as a virtual NFC tag for your credit or debit card, even if said card doesn't have an actual NFC tag inside it.

Whether you use your contactless card or a mobile payment app, every payment you make involves tokenization for extra security.

Tokenization is when your card's information is used to generate a random, temporary token for each transaction. Then, your card or mobile payment app can send that temporary token safely, rather than transmit your actual card number, name, and other sensitive information.

However you choose to pay, using a contactless payment card, a mobile payment app, or inserting your card's chip are all much safer than the old method of swiping.

Interacting with products

Traditionally RFID is used for tracking inventory in warehouses and stores. But once a product leaves the store, its RFID tag is disabled.

A lot of products now include NFC tags for additional interaction after you leave the store. Nintendo's Amiibo figures are probably the most common recent example of this:

If this isn't the cutest NFC tag, I don't know what is – Source

When you scan an Amiibo figure with your Nintendo console, you can get special characters, items, or other additional content, depending on the game and figure you use.

Your Nintendo console can also write information back to the NFC tag in your figure, again, depending on the game and figure.

Other companies like Nike have been including NFC tags in things like sports jerseys and sneakers. This allows you to get personalized content based on the product you scan (recent scores for a team, stats for a specific player, and so on), or even check that a product is genuine.

Data transfer

Unlike RFID, which is typically one-way communication between a reader and a tag, NFC allows for two-way communication.

Some phones are able to use NFC to transfer data like contacts or photos between two devices if you touch them together.

Creative uses for NFC

One of the coolest things you can do with NFC is buy a pack of tags online and program them to do different things with your phone.

For example, if you're tired of always giving your WiFi password out to guests, you could program an NFC tag to automatically connect to your network. Then, all your guests need to do is make sure NFC is enabled and hold their phones near the tag.

You could also program NFC tags to control different smart devices around your house. You could have a tag that toggles a smart lamp on or off, or one that sets the thermostat.

The commands triggered by the NFC tags can also be personalized to specific devices. For example, if you like the room a bit cooler than your partner, when you scan the thermostat tag it can lower the temperature. But when your partner scans it, it could raise the temperature to their preferred setting.

There are a lot of other interesting ways to use NFC tags to make your life just a little bit easier. Check out this video for more ideas and to see how to program your own NFC tags in both iOS and Android:

In summary

Though you might not have heard much about NFC until earlier this year (I certainly hadn't), it will likely become the standard we pay for things.

And considering all the cool stuff you can do with a phone and a pack of NFC tags, it's surprising that NFC isn't more widely adopted.

If you end up programming your own NFC tags, let me know what you did and how you did it over on Twitter.

Today roughly eight million Americans work remotely full-time. And among the most commonly held jobs include many that leverage skills honed on freeCodeCamp. These jobs include web design, software engineering, product management, and writing.

Firms are embracing the shift to hiring and empowering remote staff for many reasons. Some of the reasons frequently referenced include increased workforce productivity and access to new pools of talent.

One reason that is cited less frequently (but is seemingly as important) is that shifting to a remote labor force will significantly reduce a firm’s costs. Ultimately these cost savings will benefit shareholders.

I want to explain the cost structure that firms, and in particular technology companies, must manage. I absorbed these lessons by studying the financial documents and annual reports of companies that perhaps you aspire to work for or whose products you may use and enjoy.

I want to share these lessons with you.

First, some relevant financial context. All firms have essentially three large costs that they must manage: Property Plant and Equipment (PP&E), operating expenses, and Selling, General and Administrative (SG&A) expenses.

When a firm maintains an office (or famous large headquarters in the case of Google, Facebook, and Apple), it obviously must pay for the furniture, land, buildings, and machinery that help power its business. These fixed assets are expected to be used for at least one year and are classified as PP&E on a firm’s balance sheet.

Operating expenses are a second category of cost that firms must manage. These costs include the utilities, insurance, taxes, and payroll that a firm pays to remain open.

Lastly, a company needs to budget for SG&A expenses which are the sum of all direct or indirect selling expenses and general administrative expenses a firm accrues.

With this context in mind, we can view remote work - and the advocacy of remote work by CEOs - through a new financial light.

By placing workers in home offices (or remote co-working spaces) a firm can succeed in achieving two things in a single action: the firm can reduce its fixed and variable costs and move part of its cost structure from its balance sheet to that of its staff.

In short, companies won’t need to invest as much in buildings and equipment (i.e. lower CAPEX), which will lead directly to higher cash flow. This will lead to higher expected growth and higher stock prices over time.

Additionally, companies can decrease salaries and decrease corporate administrative costs, (i.e. lower SG&A), which will lead to higher net income to equity holders.

Facebook provides evidence of this trend.

Recently Facebook Founder and CEO Mark Zuckerberg said that as much as 50% of Facebook’s workforce could be working remotely in the next five to ten years.

What data supports this policy?

Firstly, look at the explosion of Facebook’s SG&A costs in the last three years.

• In 2019 Facebook's annual SG&A increased 80% from 2018.
• In 2018 Facebook's annual SG&A increased 56% from 2017.
• In 2017 Facebook's annual SG&A increased 32% from 2016.

Next, look at Facebook’s increase in operating expenses.

• Facebook's annual operating expenses for 2019 increased 51.05% from 2018.
• Facebook's annual operating expenses for 2018 increased 51.22% from 2017.
• Facebook's annual operating expenses for 2017 increased 34% from 2016.

Lastly, note that even as Facebook’s revenue has increased, the rate of growth is slowing.

• Facebook's annual revenue for 2019 increased by 27% from 2018.
• Facebook's annual revenue for 2018 increased 37% from 2017.
• Facebook's annual revenue for 2017 increased 47% from 2016.

Facebook’s SG&A and operating expenses are increasing faster than Facebook’s rate of revenue.

To increase profitability, Facebook will need to find ways to decrease its costs. It has a few obvious buckets to target and expanding its remote workforce can reduce PP&E, SG&A, and other operating costs in one fell swoop.

Let me explain how.

Remote workers will reduce Facebook’s dependency on furniture, land, and buildings. The decreased utilization of these fixed assets will lower Facebook's PP&E.

Secondly, remote workers will decrease Facebook’s payroll. Why? Because Facebook can pay lower wages to workers outside of dense regions where the majority of their workers are today. Hiring talent outside of metropolitan areas will enable Facebook to pay wages indexed to local costs.

A software engineer in New Hampshire or Montana will cost Facebook less to employ than that same engineer located in New York City or the Bay Area. Zuckerberg explained that Facebook “will adjust salaries to your location...that means if you live in a location where the cost of living is dramatically lower, or the cost of labor is lower, then salaries do tend to be somewhat lower in those places.”

The reduction of costs can be extrapolated to SG&A. Rent, utilities, and office supplies are three examples where Facebook can move costs from its balance sheet to that of its staff who will likely be expected, among other things, to heat and cool their home work environments, make their own coffee and food, or clean their home workspaces - at their own expense.

This shift in cost structure will likely yield disproportionate benefits to shareholders because the earnings of a firm increase when, holding all else constant, costs decrease.

What do these changes mean for you? I have a few ideas.

Remote workers would be wise to ask for and negotiate new benefits packages. Subsidized wifi or utilities seem like reasonable things to ask for if you no longer have an office to work from. Discounted home office products - a chair, desk, light, keyboard, monitor - should be included as non-taxed items when joining in a remote role. Access to virtual mentoring or mental health tools can be requested as core perks to ensure that you are growing and healthy while performing remote work.

As remote work becomes more prevalent - for full time remote workers and employees who blend their time between workplaces - new paradigms of work will arise. With this fundamental shift in the labor markets, firms will have a unique opportunity to reduce costs. How and where their expenses are cut will dictate the rate at which earnings can increase.

It seems to reason that shareholders will benefit from enhanced profitability coming from these changes. But remote engineers, designers, product managers, and sales professionals can benefit too. By asking for new and specific perks that empower remote work, these workers might see more of the upside derived from this shift in the labor market.

Many Python developers in the financial world are tasked with creating Excel documents for analysis by non-technical users.

This is actually a lot harder than it sounds. From sourcing the data to formatting the spreadsheet to deploying the final doc in a central location, there are plenty of steps involved in the process.

In this tutorial, I'm going to show you how to create Excel spreadsheets using Python that:

• Use stock market data from IEX Cloud
• Are deployed in a centralized S3 bucket so that anyone with the right URL can access them
• Automatically update daily using the cron command line utility

Step 1: Create an Account with IEX Cloud

IEX Cloud is the data provider subsidiary of the IEX stock exchange.

In case you're unfamiliar with IEX, it is an acronym for "The Investor's Exchange". IEX was founded by Brad Katsuyama to build a better stock exchange that avoids investor-unfriendly behavior like front-running and high-frequency trading. Katsuyama's exploits were famously chronicled in Michael Lewis' best-selling book Flash Boys.

I have investigated many financial data providers and IEX Cloud has the best combination of:

• High-quality data
• Affordable price

Their prices are below:

The $9/month Launch plan is plenty for many use cases.

A warning on using IEX Cloud (and any other pay-per-use data provider): it is very important that you set usage budgets from the beginning. These budgets lock you out of your account once you hit a certain dollar cost for the month.

When I first started using IEX Cloud, I accidentally created an infinite loop on a Friday afternoon that contained an API call to IEX Cloud. These API calls are priced on a cost-per-call basis...which resulted in a terrifying email from IEX:

It is a testament to IEX's customer-centricity that they agreed to reset my usage as long as I set usage budgets moving forward. Go IEX!

As with most API subscriptions, the main benefit of creating an IEX Cloud account is having an API key.

For obvious reasons, I will not share an API key in this article.

However, you can still work through this tutorial with your own API key as long as you assign it to the following variable name:

IEX_API_Key

You will see the blank IEX_API_Key variable in my code blocks throughout the rest of this tutorial.

Step 2: Write Your Python Script

Now that you have access to the API key that you'll need to gather financial data, it's time to write your Python script.

This will be the longest section of this tutorial. It is also the most flexible - we are going to create a Python script that satisfies certain pre-specified criteria, but you could modify this section to really create any spreadsheet you want!

To start, let's lay out our goal posts. We are going to write a Python script that generates an Excel file of stock market data with the following characteristics:

• It will include the 10 largest stocks in the United States
• It will contain four columns: stock ticker, company name, share price, and dividend yield.
• It will be formatted such that the header's background color is #135485 and text is white, while the spreadsheet body's background is #DADADA and the font color is black (the default).

Let's start by importing our first package.

Since spreadsheets are essentially just data structures with rows and columns, then the pandas library - including its built-in DataFrame object - is a perfect candidate for manipulating data in this tutorial.

We'll start by importing pandas under the alias pd like this:

import pandas as pd

Next, we'll specify our IEX Cloud API key. As I mentioned before, I'm not going to really include my API key, so you'll have to grab your own API key from your IEX account and include it here:

IEX_API_Key = ''

Our next step is to determine the ten largest companies in the United States.

You can answer this question with a quick Google search.

For brevity, I have included the companies (or rather, their stock tickers) in the following Python list:

tickers = [
            'MSFT',
            'AAPL',
            'AMZN',
            'GOOG',
            'FB',
            'BRK.B',
            'JNJ',
            'WMT',
            'V',
            'PG'
            ]

Next, it is time to figure out how to ping the IEX Cloud API to pull in the metrics we need for each company.

The IEX Cloud API returns JSON objects in response to HTTP requests. Since we are working with more than 1 ticker in this tutorial, we will use IEX Cloud's batch API call functionality, which allows you to request data on more than one ticker at a time. Using batch API calls has two benefits:

• It reduces the number of HTTP requests you need to make, which will make your code more performant.
• The pricing for batch API calls is slightly better with most data providers.

Here is an example of what the HTTP request might look like, with a few placeholder words where we'll need to customize the request:

https://cloud.iexapis.com/stable/stock/market/batch?symbols=TICKERS&types=ENDPOINTS&range=RANGE&token=IEX_API_Key

In this URL, we'll replace these variables with the following values:

• TICKERS will be replaced by a string that contains each of our tickers separated by a comma.
• ENDPOINTS will be replaced by a string that contains each of the IEX Cloud endpoints we want to hit, separated by a comma.
• RANGE will be replaced by 1y. These endpoints each contain point-in-time data and not time series data, so this range can really be whatever you want.

Let's put this URL into a variable called HTTP_request for us to modify later:

HTTP_request = 'https://cloud.iexapis.com/stable/stock/market/batch?symbols=TICKERS&types=ENDPOINTS&range=RANGE&token=IEX_API_Key'

Let's work through each of these variables one-by-one to determine the exact URL that we need to hit.

For the TICKERS variable, we can generate a real Python variable (and not just a placeholder word) with a simple for loop:

#Create an empty string called `ticker_string` that we'll add tickers and commas to
ticker_string = ''

#Loop through every element of `tickers` and add them and a comma to ticker_string
for ticker in tickers:
    ticker_string += ticker
    ticker_string += ','

#Drop the last comma from `ticker_string`
ticker_string = ticker_string[:-1]

Now we can interpolate our ticker_string variable into the HTTP_request variable that we created earlier using an f-string:

HTTP_request = f'https://cloud.iexapis.com/stable/stock/market/batch?symbols={ticker_string}&types=ENDPOINTS&range=RANGE&token=IEX_API_Key'

Next, we need to determine which IEX Cloud endpoints we need to ping.

Some quick investigation into the IEX Cloud documentation reveals that we only need the price and stats endpoints to create our spreadsheet.

Thus, we can replace the placeholder ENDPOINTS word from our original HTTP request with the following variable:

endpoints = 'price,stats'

Like we did with our ticker_string variable, let's substitute the endpoints variable into the ticker_string variable:

HTTP_request = f'https://cloud.iexapis.com/stable/stock/market/batch?symbols={ticker_string}&types={endpoints}&range=RANGE&token=IEX_API_Key'

The last placeholder we need to replace is RANGE. We will not replace with this a variable. Instead, we can hardcode a 1y directly into the URL path like this:

https://cloud.iexapis.com/stable/stock/market/batch?symbols={ticker_string}&types={endpoints}&range=1y&token=IEX_API_Key

We've done a lot so far, so let's recap our code base:

import pandas as pd

IEX_API_Key = ''

#Specify the stock tickers that will be included in our spreadsheet
tickers = [
            'MSFT',
            'AAPL',
            'AMZN',
            'GOOG',
            'FB',
            'BRK.B',
            'JNJ',
            'WMT',
            'V',
            'PG'
            ]


#Create an empty string called `ticker_string` that we'll add tickers and commas to
ticker_string = ''

#Loop through every element of `tickers` and add them and a comma to ticker_string
for ticker in tickers:
    ticker_string += ticker
    ticker_string += ','

#Drop the last comma from `ticker_string`
ticker_string = ticker_string[:-1]

#Create the endpoint strings
endpoints = 'price,stats'

#Interpolate the endpoint strings into the HTTP_request string
HTTP_request = f'https://cloud.iexapis.com/stable/stock/market/batch?symbols={ticker_string}&types={endpoints}&range=1y&token={IEX_API_Key}'

It is now time to ping the API and save its data into a data structure within our Python application.

We can read JSON objects with pandas' read_json method. In our case, we'll save the JSON data to a pandas DataFrame called raw_data, like this:

raw_data = pd.read_json(HTTP_request)

Let's take a moment now to make sure that the data has been imported in a nice format for our application.

If you're working through this tutorial in a Jupyter Notebook, you can simply type the name of the pandas DataFrame variable on the last line of a code cell, and Jupyter will nicely render an image of the data, like this:

As you can see, the pandas DataFrame contains a column for each stock ticker and two rows: one for the stats endpoint and one for the price endpoint. We will need to parse this DataFrame to get the four metrics we want. Let's work through the metrics one-by-one in the steps below.

Metric 1: Stock Ticker

This step is very straightforward since the stock tickers are contained in the columns of the pandas DataFrame. We can access them through the columns attribute of the pandas DataFrame like this:

raw_data.columns

To access the other metrics in raw_data, we will create a for loop that loops through each ticker in raw_data.columns. In each iteration of the loop we will add the data to a new pandas DataFrame object called output_data.

First we'll need to create output_data, which should be an empty pandas DataFrame with four columns. Here's how to do this:

output_data = pd.DataFrame(pd.np.empty((0,4)))

This creates an empty pandas DataFrame with 0 rows and 4 columns.

Now that this object has been created, here's how we can structure this for loop:

for ticker in raw_data.columns:

    #Parse the company's name - not completed yet
    company_name = ''

    #Parse the company's stock price - not completed yet
    stock_price = 0

    #Parse the company's dividend yield - not completed yet
    dividend_yield = 0


    new_column = pd.Series([ticker, company_name, stock_price, dividend_yield])
    output_data = output_data.append(new_column, ignore_index = True)

Next, let's determine how to parse the company_name variable from the raw_data object.

Metric 2: Company Name

The company_name variable is the first variable will need to be parsed from the raw_data object. As a quick recap, here's what raw_data looks like:

The company_name variable is held within the stats endpoint under the dictionary key companyName. To parse this data point out of raw_data, we can use these indexes:

raw_data[ticker]['stats']['companyName']

Including this in our for loop from before gives this:

output_data = pd.DataFrame(pd.np.empty((0,4)))

for ticker in raw_data.columns:

    #Parse the company's name - not completed yet
    company_name = raw_data[ticker]['stats']['companyName']

    #Parse the company's stock price - not completed yet
    stock_price = 0

    #Parse the company's dividend yield - not completed yet
    dividend_yield = 0


    new_column = pd.Series([ticker, company_name, stock_price, dividend_yield])
    output_data = output_data.append(new_column, ignore_index = True)

Let's move on to parsing stock_price.

Metric 3: Stock Price

The stock_price variable is contained within the price endpoint, which returns only a single value. This means we do not need to chain together indexes like we did with company_name.

Here's how we could parse stock_price from raw_data:

raw_data[ticker]['price']

Including this in our for loop gives us:

output_data = pd.DataFrame(pd.np.empty((0,4)))

for ticker in raw_data.columns:

    #Parse the company's name - not completed yet
    company_name = raw_data[ticker]['stats']['companyName']

    #Parse the company's stock price - not completed yet
    stock_price = raw_data[ticker]['price']

    #Parse the company's dividend yield - not completed yet
    dividend_yield = 0


    new_column = pd.Series([ticker, company_name, stock_price, dividend_yield])
    output_data = output_data.append(new_column, ignore_index = True)

The last metric we need to parse is dividend_yield.

Metric 4: Dividend Yield

Like company_name, dividend_yield is contained in the stats endpoint. It is held under the dividendYield dictionary key.

Here is how we could parse it out of raw_data:

raw_data[ticker]['stats']['dividendYield']

Adding this to our for loop gives us:

output_data = pd.DataFrame(pd.np.empty((0,4)))

for ticker in raw_data.columns:

    #Parse the company's name - not completed yet
    company_name = raw_data[ticker]['stats']['companyName']

    #Parse the company's stock price - not completed yet
    stock_price = raw_data[ticker]['price']

    #Parse the company's dividend yield - not completed yet
    dividend_yield = raw_data[ticker]['stats']['dividendYield']


    new_column = pd.Series([ticker, company_name, stock_price, dividend_yield])
    output_data = output_data.append(new_column, ignore_index = True)

Let's print out our output_data object to see what the data looks like:

So far so good! The next two steps are to name the columns of the pandas DataFrame and to change its index.

How to Name the Columns of a Pandas DataFrame

We can update the column names of our output_data object by creating a list of column names and assigning it to the output_data.columns attribute, like this:

output_data.columns = ['Ticker', 'Company Name', 'Stock Price', 'Dividend Yield']

Let's print out our output_data object to see what the data looks like:

Much better! Let's change the index of output_data next.

How to Change the Index of a Pandas DataFrame

The index of a pandas DataFrame is a special column that is somewhat similar to the primary key of a SQL database table. In our output_data object, we want to set the Ticker column as the DataFrame's index.

Here's how we can do this using the set_index method:

output_data.set_index('Ticker', inplace=True)

Let's print out our output_data object to see what the data looks like:

Another incremental improvement!

Next, let's deal with the missing data in output_data.

How to Handle Missing Data in Pandas DataFrames

If you take a close look at output_data, you will notice that there are several None values in the Dividend Yield column:

These None values simply indicate that the company for that row does not currently pay a dividend. While None is one way of representing a non-dividend stock, it is more common to show a Dividend Yield of 0.

Fortunately, the fix for this is quite straightforward. The pandas library includes an excellent fillna method that allows us to replace missing values in a pandas DataFrame.

Here's how we can use the fillna method to replace our Dividend Yield column's None values with 0:

output_data['Dividend Yield'].fillna(0,inplace=True)

The output_data object looks much cleaner now:

We are now ready to export our DataFrame to an Excel document! As a quick recap, here is our Python script to date:

import pandas as pd

IEX_API_Key = ''

#Specify the stock tickers that will be included in our spreadsheet
tickers = [
            'MSFT',
            'AAPL',
            'AMZN',
            'GOOG',
            'FB',
            'BRK.B',
            'JNJ',
            'WMT',
            'V',
            'PG'
            ]


#Create an empty string called `ticker_string` that we'll add tickers and commas to
ticker_string = ''

#Loop through every element of `tickers` and add them and a comma to ticker_string
for ticker in tickers:
    ticker_string += ticker
    ticker_string += ','

#Drop the last comma from `ticker_string`
ticker_string = ticker_string[:-1]

#Create the endpoint strings
endpoints = 'price,stats'

#Interpolate the endpoint strings into the HTTP_request string
HTTP_request = f'https://cloud.iexapis.com/stable/stock/market/batch?symbols={ticker_string}&types={endpoints}&range=1y&token={IEX_API_Key}'

#Create an empty pandas DataFrame to append our parsed values into during our for loop
output_data = pd.DataFrame(pd.np.empty((0,4)))

for ticker in raw_data.columns:

    #Parse the company's name
    company_name = raw_data[ticker]['stats']['companyName']

    #Parse the company's stock price
    stock_price = raw_data[ticker]['price']

    #Parse the company's dividend yield
    dividend_yield = raw_data[ticker]['stats']['dividendYield']


    new_column = pd.Series([ticker, company_name, stock_price, dividend_yield])
    output_data = output_data.append(new_column, ignore_index = True)

#Change the column names of output_data
output_data.columns = ['Ticker', 'Company Name', 'Stock Price', 'Dividend Yield']

#Change the index of output_data
output_data.set_index('Ticker', inplace=True)

#Replace the missing values of the 'Dividend Yield' column with 0
output_data['Dividend Yield'].fillna(0,inplace=True)

#Print the DataFrame
output_data

How to Export A Styled Excel Document From a Pandas DataFrame using XlsxWriter

There are multiple ways to export an xlsx file from a pandas DataFrame.

The easiest way is to use the built-in function to_excel. As an example, here's how we could export output_data to an Excel file:

output_data.to_excel('my_excel_document.xlsx)

The problem with this approach is that the Excel file has no format whatsoever. The output looks like this:

The lack of formatting in this document makes it hard to interpret.

What is the solution?

We can use the Python package XlsxWriter to generate nicely-formatted Excel files. To start, we'll want to add the following import to the beginning of our Python script:

import xlsxwriter

Next, we need to create our actual Excel file. The XlsxWriter package actually has a dedicated documentation page for how to work with pandas DataFrames, which is available here.

Our first step is to call the pd.ExcelWriter function and pass in the desired name of our xlsx file as the first argument and engine='xlsxwriter as the second argument. We will assign this to a variable called writer:

writer = pd.ExcelWriter('stock_market_data.xlsx', engine='xlsxwriter')

From there, we need to call the to_excel method on our pandas DataFrame. This time, instead of passing in the name of the file that we're trying to export, we'll pass in the writer object that we just created:

output_data.to_excel(writer, sheet_name='Sheet1')

Lastly, we will call the save method on our writer object, which saves the xlsx file to our current working directory. When all this is done, here is the section of our Python script that saves output_data to an Excel file.

writer = pd.ExcelWriter('stock_market_data.xlsx', engine='xlsxwriter')

output_data.to_excel(writer, sheet_name='Sheet1')

writer.save()

All of the formatting code that we will include in our xlsx file needs to be contained between the creation of the ExcelWriter object and the writer.save() statement.

How to Style an xlsx File Created with Python

It is actually harder than you might think to style an Excel file using Python.

This is partially because of some of the limitations of the XlsxWriter package. Its documentation states:

'XlsxWriter and Pandas provide very little support for formatting the output data from a dataframe apart from default formatting such as the header and index cells and any cells that contain dates or datetimes. In addition it isn’t possible to format any cells that already have a default format applied.

If you require very controlled formatting of the dataframe output then you would probably be better off using Xlsxwriter directly with raw data taken from Pandas. However, some formatting options are available.'

In my experience, the most flexible way to style cells in an xlsx file created by XlsxWriter is to use conditional formatting that only applies styling when a cell is not equal to None.

This has three advantages:

1. It provides more styling flexibility than the normal formatting options available in XlsxWriter.
2. You do not need to manually loop through each data point and import them into the writer object one-by-one.
3. It allows you to easily see when None values have made their way into your finalized xlsx files, since they'll be missing the required formatting.

To apply styling using conditional formatting, we first need to create a few style templates. Specifically, we will need four templates:

• One header_template that will be applied to the column names at the top of the spreadsheet
• One string_template that will be applied to the Ticker and Company Name columns
• One dollar_template that will be applied to the Stock Price column
• One percent_template that will be applied to the Dividend Yield column

Each of these format templates need to be added to the writer object in dictionaries that resemble CSS syntax. Here's what I mean:

header_template = writer.book.add_format(
        {
            'font_color': '#ffffff',
            'bg_color': '#135485',
            'border': 1
        }
    )

string_template = writer.book.add_format(
        {
            'bg_color': '#DADADA',
            'border': 1
        }
    )

dollar_template = writer.book.add_format(
        {
            'num_format':'$0.00',
            'bg_color': '#DADADA',
            'border': 1
        }
    )

percent_template = writer.book.add_format(
        {
            'num_format':'0.0%',
            'bg_color': '#DADADA',
            'border': 1
        }
    )

To apply these formats to specific cells in our xlsx file, we need to call the package's conditional_format method on writer.sheets['Stock Market Data']. Here is an example:

writer.sheets['Stock Market Data'].conditional_format('A2:B11', 
                             {
                                'type':     'cell',
                                'criteria': '<>',
                                'value':    '"None"',
                                'format':   string_template
                                }
                            )

If we generalize this formatting to the other three formats we're applying, here's what the formatting section of our Python script becomes:

writer = pd.ExcelWriter('stock_market_data.xlsx', engine='xlsxwriter')

output_data.to_excel(writer, sheet_name='Stock Market Data')

header_template = writer.book.add_format(
        {
            'font_color': '#ffffff',
            'bg_color': '#135485',
            'border': 1
        }
    )

string_template = writer.book.add_format(
        {
            'bg_color': '#DADADA',
            'border': 1
        }
    )

dollar_template = writer.book.add_format(
        {
            'num_format':'$0.00',
            'bg_color': '#DADADA',
            'border': 1
        }
    )

percent_template = writer.book.add_format(
        {
            'num_format':'0.0%',
            'bg_color': '#DADADA',
            'border': 1
        }
    )


#Format the header of the spreadsheet
writer.sheets['Stock Market Data'].conditional_format('A1:D1', 
                             {
                                'type':     'cell',
                                'criteria': '<>',
                                'value':    '"None"',
                                'format':   header_template
                                }
                            )

#Format the 'Ticker' and 'Company Name' columns
writer.sheets['Stock Market Data'].conditional_format('A2:B11', 
                             {
                                'type':     'cell',
                                'criteria': '<>',
                                'value':    '"None"',
                                'format':   string_template
                                }
                            )

#Format the 'Stock Price' column
writer.sheets['Stock Market Data'].conditional_format('C2:C11', 
                             {
                                'type':     'cell',
                                'criteria': '<>',
                                'value':    '"None"',
                                'format':   dollar_template
                                }
                            )

#Format the 'Dividend Yield' column
writer.sheets['Stock Market Data'].conditional_format('D2:D11', 
                             {
                                'type':     'cell',
                                'criteria': '<>',
                                'value':    '"None"',
                                'format':   percent_template
                                }
                            )

writer.save()

Let's take a look at our Excel document to see how its looking:

So far so good! The last incremental improvement that we can make to this document is to make its columns a bit wider.

We can specify column widths by calling the set_column method on writer.sheets['Stock Market Data'].

Here's what we'll add to our Python script to do this:

#Specify all column widths
writer.sheets['Stock Market Data'].set_column('B:B', 32)
writer.sheets['Stock Market Data'].set_column('C:C', 18)
writer.sheets['Stock Market Data'].set_column('D:D', 20)

Here's the final version of the spreadsheet:

Voila! We are good to go! You can access the final version of this Python script on GitHub here. The file is named stock_market_data.py.

Step 3: Set Up an AWS EC2 Virtual Machine to Run Your Python Script

Your Python script is finalized and ready to run.

However, we do not want to simply run this on our local machine on an ad hoc basis.

Instead, we are going to set up a virtual machine using Amazon Web Services' Elastic Compute Cloud (EC2) service.

You'll need to create an AWS account first if you do not already have one. To do this, navigate to this URL and click the "Create an AWS Account" in the top-right corner:

AWS' web application will guide you through the steps to create an account.

Once your account is created, you'll need to create an EC2 instance. This is simply a virtual server for running code on AWS infrastructure.

EC2 instances come in various operating systems and sizes, ranging from very small servers that qualify for AWS' free tier to very large servers capable of running complex applications.

We will use AWS' smallest server to run the Python script that we wrote in this article. To get started, navigate to EC2 within the AWS management console. Once you've arrived within EC2, click Launch Instance:

This will bring you to a screen that contains all of the available instance types within AWS EC2. Any machine that qualifies for AWS' free tier will be sufficient.

I chose the Amazon Linux 2 AMI (HVM):

Click Select to proceed.

On the next page, AWS will ask you to select the specifications for your machine. The fields you can select include:

• Family
• Type
• vCPUs
• Memory
• Instance Storage (GB)
• EBS-Optimized
• Network Performance
• IPv6 Support

For the purpose of this tutorial, we simply want to select the single machine that is free tier eligible. It is characterized by a small green label that looks like this:

Once you have selected a free tier eligible machine, click Review and Launch at the bottom of the screen to proceed. The next screen will present the details of your new instance for you to review. Quickly review the machine's specifications, then click Launch in the bottom right-hand corner.

Clicking the Launch button will trigger a popup that asks you to Select an existing key pair or create a new key pair. A key pair is comprised of a public key that AWS holds and a private key that you must download and store within a .pem file. You must have access to that .pem file in order to access your EC2 instance (typically via SSH). You also have the option to proceed without a key pair, but this is not recommended for security reasons.

Once you have selected or created a key pair for this EC2 instance and click the radio button for I acknowledge that I have access to the selected private key file (data-feeds.pem), and that without this file, I won't be able to log into my instance, you can click Launch Instances to proceed.

Your instance will now begin to launch. It can take some time for these instances to boot up, but once its ready, its Instance State will show as running in your EC2 dashboard.

Next, you will need to push your Python script into your EC2 instance. Here is a generic command state statement that allows you to move a file into an EC2 instance:

scp -i path/to/.pem_file path/to/file   username@host_address.amazonaws.com:/path_to_copy

Run this statement with the necessary replacements to move stock_market_data.py into the EC2 instance.

Trying to run stock_market_data.py at this point will actually result in an error because the EC2 instance does not come with the necessary Python packages.

To fix this, you can either export a requirements.txt file and import the proper packages using pip, or you can simply run the following:

sudo yum install python3-pip
pip3 install pandas
pip3 install xlsxwriter

Once this is done, you can SSH into the EC2 instance and run the Python script from the command line with the following statement:

python3 stock_market_data.py

Step 4: Create an AWS S3 Bucket to Hold the Finished Python Script

With the work that we have completed so far, our Python script can be executed inside of our EC2 instance.

The problem with this is that the xlsx file will be saved to the AWS virtual server.

It is not accessible to anyone but us in that server, which limits its usefulness.

To fix this, we are going to create a public bucket on AWS S3 where we can save the xlsx file. Anyone who has the right URL will be able to download this file once this change is made.

To start, navigate to AWS S3 from within the AWS Management Console. Click Create bucket in the top right:

On the next screen, you will need to pick a name for your bucket and an AWS region for the bucket to be hosted in. The bucket name must be unique and cannot contain spaces or uppercase letters. The region does not matter much for the purpose of this tutorial, so I will be using the default region of US East (Ohio) us-east-2).

You will need to change the Public Access settings in the next section to match this configuration:

Click Create bucket to create your bucket and conclude this step of this tutorial!

Step 5: Modify Your Python Script to Push the xlsx File to AWS S3

Our AWS S3 bucket is now ready to hold our finalized xlsx document. We will now make a small change to our stock_market_data.py file to push the finalized document to our S3 bucket.

We will need to use the boto3 package to do this. boto3 is the AWS Software Development Kit (SDK) for Python, allowing Python developers to write software that connects to AWS services. To start, you'll need to install boto3 on your EC2 virtual machine. Run the following command line statement to do this:

pip3 install boto3

You will also need to import the library into stock_market_data.py by adding the following statement to the top of the Python script.

import boto3

We will need to add a few lines of code to the end of stock_market_data.py to push the final document to AWS S3.

s3 = boto3.resource('s3')
s3.meta.client.upload_file('stock_market_data.xlsx', 'my-S3-bucket', 'stock_market_data.xlsx', ExtraArgs={'ACL':'public-read'})

The first line of this code, s3 = boto3.resource('s3'), allows our Python script to connect to Amazon Web Services.

The second line of code calls a method from boto3 that actually uploads our file to S3. It takes four arguments:

• stock_market_data.xlsx - the name of the file on our local machine.
• my-S3-bucket - the name of the S3 bucket that we're uploading our file to.
• stock_market_data.xlsx - the desired name of the file within the S3 bucket. In most cases, this will have the same value as the first argument passed into this method.
• ExtraArgs={'ACL':'public-read'} - this is an optional argument that tells AWS to make the uploaded file publicly-readable.

Step 6: Schedule Your Python Script to Run Periodically Using Cron

So far, we have completed the following:

• Built our Python script
• Created an EC2 instance and deployed our code there
• Created an S3 bucket where we can push the final xlsx document
• Modified the original Python script to upload the finalized stock_market_data.xlsx file to an AWS S3 bucket

The only step that is left is to schedule the Python script to run periodically.

We can do this using a command-line utility called cron. To start, we will need to create a cron expression that tells the utility when to run the code. The crontab guru website is an excellent resource for this.

Here's how you can use crontab guru to get cron expression that means every day at noon:

Now we need to instruct our EC2 instance's cron daemon to run stock_market_data.py at this time each day.

To do this, we will first create a new file in our EC2 instance called stock_market_data.cron.

Open up this file and type in our cron expression followed by the statement that should be executed at the command line at that specified time.

Our command line statement is python3 stock_market_data.py, so here is what should be contained in stock_market_data.cron:

00 12 * * * python3 stock_market_data.py

If you run an ls command in your EC2 instance, you should now see two files:

stock_market_data.py    stock_market_data.cron

The last step of this tutorial is to load stock_market_data.cron into the crontab. You can think of the crontab as a file that contains commands and instructions for the cron daemon to execute. In other words, the crontab contains batches of cron jobs.

First, let's see what's in our crontab. It should be empty since we have not put anything in it! You can view the contents of your crontab with the following command:

crontab -l

To load stock_market_data.cron into the crontab, run the following statement on the command line:

crontab stock_market_data.cron

Now when you run crontab -l, you should see:

00 12 * * * python3 stock_market_data.py

Our stock_market_data.py script will now run at noon every day on our AWS EC2 virtual machine!

Final Thoughts

In this article, you learned how to create automatically-updating Excel spreadsheets of financial data using Python, IEX Cloud, and Amazon Web Services.

Here are the specific steps we covered in this tutorial:

• How to create an account with IEX Cloud
• How to write a Python script that generates beautiful Excel documents using pandas and XlsxWriter
• How to launch an AWS EC2 instance and deploy code on it
• How to create an AWS S3 bucket
• How to push files to an AWS S3 bucket from within a Python script
• How to schedule code to run using the cron software utility

This article was published by Nick McCullum, who teaches people how to code on his website.

An introduction to web scraping for finance

Ever wished you could access historical options data, but got blocked by a paywall? What if you just want it for research, fun, or to develop a personal trading strategy?

In this tutorial, you’ll learn how to use Python and BeautifulSoup to scrape financial data from the Web and build your own dataset.

Getting Started

You should have at least a working knowledge of Python and Web technologies before beginning this tutorial. To build these up, I highly recommend checking out a site like codecademy to learn new skills or brush up on old ones.

First, let’s spin up your favorite IDE. Normally, I use PyCharm but, for a quick script like this Repl.it will do the job too. Add a quick print (“Hello world”) to ensure your environment is set up correctly.

Now we need to figure out a data source.

Unfortunately, Cboe’s awesome options chain data is pretty locked down, even for current delayed quotes. Luckily, Yahoo Finance has solid enough options data here. We’ll use it for this tutorial, as web scrapers often need some content awareness, but it is easily adaptable for any data source you want.

Dependencies

We don’t need many external dependencies. We just need the Requests and BeautifulSoup modules in Python. Add these at the top of your program:

from bs4 import BeautifulSoupimport requests

Create a main method:

def main():  print(“Hello World!”)if __name__ == “__main__”:  main()

Scraping HTML

Now you’re ready to start scraping! Inside main(), add these lines to fetch the page’s full HTML:

data_url = “https://finance.yahoo.com/quote/SPY/options"data_html = requests.get(data_url).contentprint(data_html)

This fetches the page’s full HTML content, so we can find the data we want in it. Feel free to give it a run and observe the output.

Feel free to comment out print statements as you go — these are just there to help you understand what the program is doing at any given step.

BeautifulSoup is the perfect tool for working with HTML data in Python. Let’s narrow down the HTML to just the options pricing tables so we can better understand it:

 content = BeautifulSoup(data_html, “html.parser”) # print(content)

 options_tables = content.find_all(“table”) print(options_tables)

That’s still quite a bit of HTML — we can’t get much out of that, and Yahoo’s code isn’t the most friendly to web scrapers. Let’s break it down into two tables, for calls and puts:

 options_tables = [] tables = content.find_all(“table”) for i in range(0, len(content.find_all(“table”))):   options_tables.append(tables[i])

 print(options_tables)

Yahoo’s data contains options that are pretty deep in- and out-of-the-money, which might be great for certain purposes. I’m only interested in near-the-money options, namely the two calls and two puts closest to the current price.

Let’s find these, using BeautifulSoup and Yahoo’s differential table entries for in-the-money and out-of-the-money options:

expiration = datetime.datetime.fromtimestamp(int(datestamp)).strftime(“%Y-%m-%d”)

calls = options_tables[0].find_all(“tr”)[1:] # first row is header

itm_calls = []otm_calls = []

for call_option in calls:    if “in-the-money” in str(call_option):  itm_calls.append(call_option)  else:    otm_calls.append(call_option)

itm_call = itm_calls[-1]otm_call = otm_calls[0]

print(str(itm_call) + “ \n\n “ + str(otm_call))

Now, we have the table entries for the two options nearest to the money in HTML. Let’s scrape the pricing data, volume, and implied volatility from the first call option:

 itm_call_data = [] for td in BeautifulSoup(str(itm_call), “html.parser”).find_all(“td”):   itm_call_data.append(td.text)

print(itm_call_data)

itm_call_info = {‘contract’: itm_call_data[0], ‘strike’: itm_call_data[2], ‘last’: itm_call_data[3],  ‘bid’: itm_call_data[4], ‘ask’: itm_call_data[5], ‘volume’: itm_call_data[8], ‘iv’: itm_call_data[10]}

print(itm_call_info)

Adapt this code for the next call option:

# otm callotm_call_data = []for td in BeautifulSoup(str(otm_call), “html.parser”).find_all(“td”):  otm_call_data.append(td.text)

# print(otm_call_data)

otm_call_info = {‘contract’: otm_call_data[0], ‘strike’: otm_call_data[2], ‘last’: otm_call_data[3],  ‘bid’: otm_call_data[4], ‘ask’: otm_call_data[5], ‘volume’: otm_call_data[8], ‘iv’: otm_call_data[10]}

print(otm_call_info)

Give your program a run!

You now have dictionaries of the two near-the-money call options. It’s enough just to scrape the table of put options for this same data:

puts = options_tables[1].find_all("tr")[1:]  # first row is header

itm_puts = []  otm_puts = []

for put_option in puts:    if "in-the-money" in str(put_option):      itm_puts.append(put_option)    else:      otm_puts.append(put_option)

itm_put = itm_puts[0]  otm_put = otm_puts[-1]

# print(str(itm_put) + " \n\n " + str(otm_put) + "\n\n")

itm_put_data = []  for td in BeautifulSoup(str(itm_put), "html.parser").find_all("td"):    itm_put_data.append(td.text)

# print(itm_put_data)

itm_put_info = {'contract': itm_put_data[0],                  'last_trade': itm_put_data[1][:10],                  'strike': itm_put_data[2], 'last': itm_put_data[3],                   'bid': itm_put_data[4], 'ask': itm_put_data[5], 'volume': itm_put_data[8], 'iv': itm_put_data[10]}

# print(itm_put_info)

# otm put  otm_put_data = []  for td in BeautifulSoup(str(otm_put), "html.parser").find_all("td"):    otm_put_data.append(td.text)

# print(otm_put_data)

otm_put_info = {'contract': otm_put_data[0],                  'last_trade': otm_put_data[1][:10],                  'strike': otm_put_data[2], 'last': otm_put_data[3],                   'bid': otm_put_data[4], 'ask': otm_put_data[5], 'volume': otm_put_data[8], 'iv': otm_put_data[10]}

Congratulations! You just scraped data for all near-the-money options of the S&P 500 ETF, and can view them like this:

 print("\n\n") print(itm_call_info) print(otm_call_info) print(itm_put_info) print(otm_put_info)

Give your program a run — you should get data like this printed to the console:

{‘contract’: ‘SPY190417C00289000’, ‘last_trade’: ‘2019–04–15’, ‘strike’: ‘289.00’, ‘last’: ‘1.46’, ‘bid’: ‘1.48’, ‘ask’: ‘1.50’, ‘volume’: ‘4,646’, ‘iv’: ‘8.94%’}{‘contract’: ‘SPY190417C00290000’, ‘last_trade’: ‘2019–04–15’, ‘strike’: ‘290.00’, ‘last’: ‘0.80’, ‘bid’: ‘0.82’, ‘ask’: ‘0.83’, ‘volume’: ‘38,491’, ‘iv’: ‘8.06%’}{‘contract’: ‘SPY190417P00290000’, ‘last_trade’: ‘2019–04–15’, ‘strike’: ‘290.00’, ‘last’: ‘0.77’, ‘bid’: ‘0.75’, ‘ask’: ‘0.78’, ‘volume’: ‘11,310’, ‘iv’: ‘7.30%’}{‘contract’: ‘SPY190417P00289000’, ‘last_trade’: ‘2019–04–15’, ‘strike’: ‘289.00’, ‘last’: ‘0.41’, ‘bid’: ‘0.40’, ‘ask’: ‘0.42’, ‘volume’: ‘44,319’, ‘iv’: ‘7.79%’}

Setting up recurring data collection

Yahoo, by default, only returns the options for the date you specify. It’s this part of the URL: https://finance.yahoo.com/quote/SPY/options?date=1555459200

This is a Unix timestamp, so we’ll need to generate or scrape one, rather than hardcoding it in our program.

Add some dependencies:

import datetime, time

Let’s write a quick script to generate and verify a Unix timestamp for our next set of options:

def get_datestamp():  options_url = “https://finance.yahoo.com/quote/SPY/options?date="  today = int(time.time())  # print(today)  date = datetime.datetime.fromtimestamp(today)  yy = date.year  mm = date.month  dd = date.day

The above code holds the base URL of the page we are scraping and generates a datetime.date object for us to use in the future.

Let’s increment this date by one day, so we don’t get options that have already expired.

dd += 1

Now, we need to convert it back into a Unix timestamp and make sure it’s a valid date for options contracts:

 options_day = datetime.date(yy, mm, dd) datestamp = int(time.mktime(options_day.timetuple())) # print(datestamp) # print(datetime.datetime.fromtimestamp(options_stamp))

 # vet timestamp, then return if valid for i in range(0, 7):   test_req = requests.get(options_url + str(datestamp)).content   content = BeautifulSoup(test_req, “html.parser”)   # print(content)   tables = content.find_all(“table”)

 if tables != []:   # print(datestamp)   return str(datestamp) else:   # print(“Bad datestamp!”)   dd += 1   options_day = datetime.date(yy, mm, dd)   datestamp = int(time.mktime(options_day.timetuple()))  return str(-1)

Let’s adapt our fetch_options method to use a dynamic timestamp to fetch options data, rather than whatever Yahoo wants to give us as the default.

Change this line:

data_url = “https://finance.yahoo.com/quote/SPY/options"

To this:

datestamp = get_datestamp()data_url = “https://finance.yahoo.com/quote/SPY/options?date=" + datestamp

Congratulations! You just scraped real-world options data from the web.

Now we need to do some simple file I/O and set up a timer to record this data each day after market close.

Improving the program

Rename main() to fetch_options() and add these lines to the bottom:

options_list = {‘calls’: {‘itm’: itm_call_info, ‘otm’: otm_call_info}, ‘puts’: {‘itm’: itm_put_info, ‘otm’: otm_put_info}, ‘date’: datetime.date.fromtimestamp(time.time()).strftime(“%Y-%m-%d”)}return options_list

Create a new method called schedule(). We’ll use this to control when we scrape for options, every twenty-four hours after market close. Add this code to schedule our first job at the next market close:

from apscheduler.schedulers.background import BackgroundScheduler

scheduler = BackgroundScheduler()

def schedule():  scheduler.add_job(func=run, trigger=”date”, run_date = datetime.datetime.now())  scheduler.start()

In your if __name__ == “__main__”: statement, delete main() and add a call to schedule() to set up your first scheduled job.

Create another method called run(). This is where we’ll handle the bulk of our operations, including actually saving the market data. Add this to the body of run():

  today = int(time.time()) date = datetime.datetime.fromtimestamp(today) yy = date.year mm = date.month dd = date.day

 # must use 12:30 for Unix time instead of 4:30 NY time next_close = datetime.datetime(yy, mm, dd, 12, 30)

 # do operations here “”” This is where we’ll write our last bit of code. “””

 # schedule next job scheduler.add_job(func=run, trigger=”date”, run_date = next_close)

 print(“Job scheduled! | “ + str(next_close))

This lets our code call itself in the future, so we can just put it on a server and build up our options data each day. Add this code to actually fetch data under “”” This is where we’ll write our last bit of code. “””

options = {}

 # ensures option data doesn’t break the program if internet is out try:   if next_close > datetime.datetime.now():     print(“Market is still open! Waiting until after close…”)   else:     # ensures program was run after market hours     if next_close < datetime.datetime.now():      dd += 1       next_close = datetime.datetime(yy, mm, dd, 12, 30)       options = fetch_options()       print(options)       # write to file       write_to_csv(options)except:  print(“Check your connection and try again.”)

Saving data

You may have noticed that write_to_csv isn’t implemented yet. No worries — let’s take care of that here:

def write_to_csv(options_data):  import csv  with open(‘options.csv’, ‘a’, newline=’\n’) as csvfile:  spamwriter = csv.writer(csvfile, delimiter=’,’)  spamwriter.writerow([str(options_data)])

Cleaning up

As options contracts are time-sensitive, we might want to add a field for their expiration date. This capability is not included in the raw HTML we scraped.

Add this line of code to save and format the expiration date towards the top of fetch_options():

expiration =  datetime.datetime.fromtimestamp(int(get_datestamp())).strftime("%Y-%m-%d")

Add ‘expiration’: expiration to the end of each option_info dictionary like so:

itm_call_info = {'contract': itm_call_data[0],  'strike': itm_call_data[2], 'last': itm_call_data[3],   'bid': itm_call_data[4], 'ask': itm_call_data[5], 'volume': itm_call_data[8], 'iv': itm_call_data[10], 'expiration': expiration}

Give your new program a run — it’ll scrape the latest options data and write it to a .csv file as a string representation of a dictionary. The .csv file will be ready to be parsed by a backtesting program or served to users through a webapp. Congratulations!

The recent tax reform bill passed in the US has raised a lot of questions about wealth distribution in the country. While there’s been a lot of focus on how the tax plan will impact income, there’s been less attention focused on how this plan impacts the assets of wealthy households.

The goal of this post is to show how the R programming language can be used to data mine publicly available sources to better understand the net worth of affluent households in the US. Using data from the 2016 Federal Reserve Survey of Consumer Finances, we investigate the following questions:

• How rich are the top 1% and top 0.1% of households?
• Are there different types of millionaires in the US?
• How do asset allocations differ across different net worth segments?

To answer these questions, we present descriptive statistics of this survey data and perform cluster analysis on affluent households, which we identify as households with a net worth of more than $1,000,000 USD.

Based on the survey data, our analysis shows that the net worth of the top 1% of households in the US is $10.4M and the net worth of the top 0.1% of households is $43.2M. This post presents an analysis of the different asset compositions of millionaires, and shows how asset allocations differ between the top 10%, 1%, and 0.1% of households in the US. The R source code used to produce all results and figures presented in this post is available as a Jupyter Notebook.

Setting Up the Environment
To perform cluster analysis on the affluent households in the US, we use several packages available from the CRAN library for R. For exploratory data analysis, we like to use the R kernel for the Jupyter notebook, since it enables data scientists to easily store notebooks on GitHub and share findings with other teams.

Setting up this environment is outside the scope of this post, but I’ve previously detailed our motivation for this setup in this post, and additional details for setting up Jupyter with R support are available here.

We’re now ready to start digging into the survey data to better understand the assets of affluent households in the US. To begin, we’ll load several libraries that will help us analyze the survey data and perform clustering.

The code block below shows the libraries that need to be loaded for executing this notebook. The readxl library is needed to read the source data and convert it into a data frame, the reldist and ENmisc libraries are used for computing distributions with weighted data sets, and the remaining libraries are used for cluster analysis.

library(readxl)     # for reading xlsx files library(reldist)    # for computing weighted statisticslibrary(ENmisc)     # for weighted box plotslibrary(plotly)     # for interactive plotslibrary(factoextra) # for factor mapslibrary(FactoMineR) # Principal Component Analysis (PCA)library(cluster)    # Clustering algorithms (CLARA)library(class)      # for KNN

Getting the Data
The next step is to download the data from the Federal Reserve website. The survey data is available as a zipped xlsx file. To download the data and load it into R as a data frame for analysis, we use the code block below. Since this is a large file, we make sure that we do not download it multiple times. The resulting file that we unzip is about 40MB and takes some time to load into a data frame.

if (!file.exists("SCFP2016.xlsx")) {          download.file(   "https://www.federalreserve.gov/econres/files/scfp2016excel.zip",   "SCFP2016.zip")  unzip("scfp2016.zip")}

df <- read_excel("SCFP2016.xlsx")

Summary Statistics
Now that we have the survey data loaded into R, we can start to analyze the asset allocation of the richest households in the US. We’ll start with some summary stats: how many survey participants and how many households are represented by this survey? Counting the number of rows in the data frame answers the first questions (31.2k), and adding up the weights of all of the survey respondents answers the second question (126M).

# How many survey participants?nrow(df)

# How many households does the survey represent?floor(sum(df$WGT)))

# What is the weighted mean of household net worth?floor(sum(df$NETWORTH*df$WGT)/sum(df$WGT)))

# what is the median NW in US?  reldist::wtd.quantile(df$NETWORTH, q=0.5, weight = df$WGT)

# who is the 1%? reldist::wtd.quantile(df$NETWORTH, q=0.99, weight = df$WGT)

# top 0.1% Ultra-high net worth households  reldist::wtd.quantile(df$NETWORTH, q=0.999, weight = df$WGT)

To answer questions about averages, such as what is the mean household net worth, we need to use weighted statistics (since the weight of one survey respondent can be much larger than others). To compute the mean net worth, we can use built-in R functions, which returns a value of $690k. However, since net worth is much closer to a log normal than normal distribution, we should use other approaches.

To compute the median value with weighted responses, we use the reldist library, which assigns more support to respondents with larger weights and less support to respondents with lower weights. When using this approach to compute the weighted median, we find that the 50th percentile for household net worth in the US is $97k.

The net worth of the top 1% is $10.4M and the net worth of the top 0.1% is $43.2M. We use the wtd.quantile function to compute these descriptive statistics, and the code sample above uses the fully quantified function names, due to collisions with the ENmisc package.

Demographic Data
The survey data provides a number of different demographic variables that can be used to analyze net worth by different factors. These variables include race, marital status, education level, employment status, and others. The goal of this post is to show how asset allocation varies by net worth segment, and analysis of how these demographic factors impact net worth is left as an exercise for the reader.

# filter on ages 30 - 84, and group into 5-year bucketsdata <- df[df$AGE >= 30 & df$AGE < 85, ]ages <- floor(data$AGE/5.0)*5

# plot the weighted box plotwtd.boxplot(log10(1 + data$NETWORTH) ~ ages, data = data,   weights = data$WGT, main = 'Net Worth by Age',  xlab="Age", ylab="Net Worth (Log10)")

One demographic variable that we did explore is the impact of age on household net worth. As expected, the median net worth does increase as the head of household becomes older, with net worth plateauing around 60 years old.

The code above shows how to display a box plot of the survey data by age, using the ENmisc package to compute weighted distributions. The results of this plot show that the median net worth of households in the US are $114k at 40, $163k at 50, and $243k at 60.

Asset Allocation
The next step in the notebook is to evaluate the asset allocation of different net worth segments. For this analysis, we define a segment based on the log10 value of the household net worth. That means all 5-figure households get grouped together, all 6-figure households get grouped together, as so on.

# normalize assets by total financial and non-financial amountshouseholds <- data.frame(  LIQ = df$LIQ/assets,  ...  BUS = df$BUS/assets,    OTHNFIN = df$OTHNFIN/assets

# split into net worth segments, and compute mean distributions nw <- floor(log10(households$netWorth))segment <- ifelse(nw == 4, "     $10k",       ifelse(nw == 5, "    $100K",       ifelse(nw == 6, "   $1M",                     ifelse(nw == 7, "  $10M",                     ifelse(nw == 8, " $100M", "$1B+")))))

results <- as.data.frame((aggregate(households,list(segment),mean)))

# plot the resultsplot <- plot_ly(results, x = ~Group.1, y = ~100*LIQ, type = 'bar', name = 'Liquid') %>%  add_trace(y = ~100*CDS, name = 'Certificates of Deposit') %>%  add_trace(y = ~100*NMMF, name = 'Mutual Funds') %>%  ...  add_trace(y = ~100*BUS, name = 'Business Interests') %>%  layout(yaxis = list(title = '% of Assets',     ticksuffix = "%"), xaxis = list(title = "Net Worth"),    title = "Asset Allocation by Net Worth", barmode = 'stack')

The code above computes the asset allocation by household, by dividing the amount of assets for a household, such as business equity (BUS) over the total number of financial and non-financial assets, excluding debt. The “…” pattern is used to indicate that multiple rows have been excluded from the code snippet that are listed in the full notebook. The second code block groups the households into different net worth segments, and the third block plots the results as shown below.

The results show that the wealthy, ultra wealthy, and billionaires have large amounts of assets in business equity (stock and options). The wealthy have only a small percentage of assets in retirement funds, and instead have assets in stock, mutual funds, and residential and commercial real estate.

Clustering Millionaires
So far, we’ve looked at aggregate statistics of wealthy households, but that doesn’t tell us about the different types of affluent households. To understand how assets vary across affluent homes, we can use cluster analysis. One of the most useful ways to visualize the difference between instances in a sample population is to use factor maps to visualize the variance in the population.

# filter on affluent households, and print the total numberaffluent <- households[households$netWorth >= 1000000, ]cat(paste("Affluent Households: ", floor(sum(affluent$weight))))

# plot a Factor Map of assets fviz_pca_var(PCA(affluent,  graph = FALSE), col.var="contrib",      gradient.cols = c("#00AFBB", "#E7B800", "#FC4E07"), repel = TRUE)+  labs(title ="Affluent Households - Assets Factor Map")

The code above first filters survey respondents to affluent households, with a net worth of more than $1M USD, and then plots a factor map using principal component analysis (PCA). The figure below shows how the different assets impact the trajectory of plotting a household across the two principal components discovered via PCA.

The results plotted below show that there are a few different assets groups that vary across affluent net worth. The most significant factor is business equity. Some other groupings of factors include investment assets (STOCKS, BONDS) and real estate assets/retirement funds.

How many clusters to use?
We’ve now shown signs that there are different types of millionaires, and that assets vary based on net worth segments. To understand how asset allocation differs by net worth segment, we can use cluster analysis. We first identify clusters in the affluent survey respondents, and then apply these labels to the overall population of survey respondents.

k <- 7res.hc <- eclust(households[sample(nrow(households), 1000), ],  "hclust", k = k, graph = FALSE) fviz_dend(res.hc, rect = TRUE, show_labels = FALSE)

To determine how many clusters to use, we created a cluster dendrogram using the code snippet above, shown as the header image of this post. We also varied the number of clusters, k, until we had the largest number of distinctly identifiable clusters.

If you’d prefer to take a quantitative approach, you can use the _fviznbclust function, which computes the optimal number of clusters using a silhouette metric. For our analysis, we decided to use 7 clusters.

clarax <- clara(affluent, k)fviz_cluster(clarax, stand = FALSE, geom = "point", ellipse = F)

To cluster the affluent households into unique groupings, we used the CLARA algorithm. A visualization of the different clusters is shown below. The results are similar to PCA and the factor map approach discussed above.

Cluster Descriptions
Now that we’ve determined how many clusters to use, it’s useful to inspect the clusters and assign qualitative labels based on the feature sets. The code snippet below shows how to compute the average feature values for the 7 different clusters.

groups <- clarax$clusteringresults <- as.data.frame(t(aggregate(affluent,list(groups),mean)))  results[2:18,]

The results of this code block are shown below. Based on these results, we came up with the following cluster descriptions:

• V1: Stocks/Bonds — 31% of assets, followed by home and mutual funds
• V2: Diversified — 53% busequity, 10% home and 9% in other real estate
• V3: Residential Real Estate — 48% of assets
• V4: Mutual Funds — 50% of assets
• V5: Retirement — 48% of assets
• V6: Business Equity — 85% of assets
• V7: Commercial Real Estate — 59% of assets

With the exception of cluster V7, containing only 3% of the population, most of the clusters are relatively even in size. The second smallest cluster represents 12% of the population while the largest cluster represents 20%. You can use table(groups) to show the unweighted cluster population sizes.

Cluster Populations by Net Worth Segments
The last step in this analysis is to apply the different cluster assignments to the overall population, and to group the populations by net worth segments. Since we trained the clusters on only affluent households, we need to use a classification algorithm to label the non-affluent households in the population. The code snippet below uses knn to accomplish this task.

The remaining code blocks compute the number of households that are classified as each cluster, for each of the net worth segments.

# assign all of the households to a cluster groups <- knn(train = affluent, test = households,   cl = clarax$clustering, k = k, prob = T, use.all = T)

# figure out how many households are in each cluster clusters <- data.frame(  c1 = ifelse(groups == 1, weights, 0),   ...   c7 = ifelse(groups == 7, weights, 0) )

# assign each household to a net worth cluster nw <- floor(2*log10(nwHouseholds))/2results <- as.data.frame(t(aggregate(clusters,list(nw),sum)))

# compute the number of households that belong to each segmentresults$V1 <- results$V1/sum(ifelse(nw == 4, weights, 0))...results$V11 <- results$V11/sum(ifelse(nw == 9, weights, 0))

# plot the results plot <- plot_ly(results, x = ~10^Group.1, y = ~100*c1, type = 'scatter', mode = 'lines', name = "Stocks") %>%  add_trace(y = ~100*c2, name = "Diversified") %>%  ...  add_trace(y = ~100*c7, name = "Commercial R.E.") %>%  layout(yaxis = list(title = '% of Households', ticksuffix = "%"),     xaxis = list(title = "Net Worth ($)", type = "log"),      title  = "Cluster Populations by Net Worth")

The results of this process are shown in the figure below. The chart shows some obvious and some novel results: home ownership and retirement funds make up the majority of assets for non-affluent households, there is a relatively even mix of clusters around $2M (excluding commercial real estate and business equity), and business equity dominates net worth for the ultra-wealthy households, followed by other investment assets.

Summary
In this post we used R to download and analyze data from the 2016 Federal Reserve survey of consumer finances to understand how wealthy the top households are in the US, and to cluster affluent households by asset allocation. We identified 7 distinct groups of millionaires, and showed how the distribution of clusters varies based on net worth segment. Please keep in mind that the results presented are from weighted survey data, and may not be representative of the overall US population.

Ben Weber is a principal data scientist at Zynga. We are hiring!

Bitcoin (BTC) recently smashed an all-time high of $11,400 USD and subsequently dropped to as low as $8,595 within a few hours. It’s incredibly important to not get lost in the pandemonium and to stay informed about how Bitcoin is progressing technologically.

Anybody considering buying Bitcoin should at the very least learn two things:

1. the history of the technology behind Bitcoin
2. and more importantly, what lies ahead in Bitcoin’s future.

As Bitcoin expert Andreas M. Antonopoulos says, “Invest in education instead of speculation.”

With Bitcoin and its underlying blockchain being such incredibly new technological concepts, it can seem daunting at times to try and research and understand its underlying technical details. This article is written in an effort to highlight the scalability problem Bitcoin faces, and what expected or proposed solutions to that problem are. There are some really exciting ones out there that this article discusses!

I wrote this as I was learning about Bitcoin myself, to aggregate the great amount of information about Bitcoin’s future from countless sources out there. When writing this, I kept in mind even those who don’t possess a programming background. However, it is assumed that the reader has a very basic understanding of Bitcoin as a currency and what a blockchain is. Coindesk has a great 5-minute read that should get you up to speed here if you’re just starting out with Bitcoin or if you need a refresher.

Let’s begin with the scalability problem Bitcoin faces.

The Transaction Throughput Problem

When Bitcoin was first introduced to the world, its creator Satoshi Nakomoto described Bitcoin in the Bitcoin whitepaper as “A purely peer-to-peer version of electronic cash would allow online payments to be sent directly from one party to another without going through a financial institution.”

One of Bitcoin’s fundamental values was instant and secure peer-to-peer payment transactions. Now, more than ever, Bitcoin is emerging as the prevailing cryptocurrency in the global market, with a 1,200%+ increase in value over the last year alone.

Because of this unprecedented growth, the number of transactions on the Bitcoin blockchain has also increased, with up to 400,000 transactions per day being conducted. This rapid increase in transactions is posing to be a serious scalability problem for the blockchain, with over 90,000 transactions being backlogged as unconfirmed at the moment.

As Bitcoin’s price skyrockets, so does its usage. Note the stagnation at around 400,000 transactions per day

In order to understand why transactions are being backlogged, Bitcoin transactions must first be explained.

Every time a user sends a Bitcoin transaction from his or her wallet to another, the transaction is added into the memory pool (mempool), which is essentially a pool of all unconfirmed transactions in the Bitcoin network. This pool is upheld by individual memory pools on machines that also hold a copy of the blockchain ledger, called nodes.

From the mempool, miners select transactions that they want to verify. Once miners validate a transaction (i.e. confirm that the sender actually has enough bitcoins to send to the receiver), they add it to a new block, which is eventually published to the blockchain. Other nodes then iterate through this newly published block’s transactions to ensure the block is valid, before accepting the block as a part of its ledger.

_A diagram of the transaction lifecycle, source:[ weusecoins.com](https://www.weusecoins.com/en/questions/" rel="noopener" target="blank" title=")

Let’s calculate the throughput of transactions:

• The median transaction size approximately 250 bytes
• A block’s size is limited to 1MB (1000000 bytes)
• Thus, a block holds around 4000 transactions (1MB divided by 250 bytes)
• A block can only be published to the blockchain once every 10 minutes on average (600 seconds).
• 4000 transactions (at most) are published every 600 seconds, at a rate of 6.66 transactions / second

With over 90,000 unconfirmed transactions in the mempool, how does a miner select which transactions to verify? Transaction fees! The sender of a transaction has the option of adding a custom transaction fee to its transaction intended for the miner, incentivizing a miner to select the transaction and have it verified faster. Miners will select the transactions that have the highest fee attached to them to maximize profits. Theoretically, you can send a transaction with no fee. But if there are transactions that have fees higher than yours in the pool, why would yours ever get picked?

As Bitcoin’s user base grows, so does the average transaction fee. At most, there are only 7 transactions that are processed every second and everyone wants to get their transaction verified first. At the moment, the average transaction fee is approximately $3.58 USD. This fee is certainly not ideal — if you want to send your friend a couple of dollars worth of bitcoin, you may end up spending more in transaction fees than the transaction value itself! Therein lies the problem, and if all else remains equal, transaction fees can be expected to rise due to the transaction bottleneck.

Solving the Throughput Problem

A proposed solution to this bottleneck that has brought great controversy to the Bitcoin community is to simply raise the block size from the original 1MB limit, thus allowing more transactions per block.

Every time the block size is increased in the chain, a hard fork is required, meaning an entirely new copy of the chain must be created, therefore requiring consensus from the Bitcoin community. Because millions of people use Bitcoin, gaining consensus is difficult and efforts should be made to avoid it. Furthermore, although the block size can be increased enough to accommodate the current backlog of transactions, as Bitcoin’s userbase continues to grow, there will eventually be another backlog of unconfirmed transactions, so another block size increase will be needed, and subsequently another hard fork.

So why don’t we just make the block size large enough to ensure the throughput will never be a bottleneck, no matter how many people are using it? First, the mathematics of a block size even remotely large enough to handle mass adoption are impractical and will restrict mining to incredibly powerful machines that only large corporations will be able to maintain, introducing an element of centralization.

_Given a block size big enough for just one city’s population, it would restrict node hosting and mining to only those with the most powerful machines, i.e. massive businesses, source: [Lightning Network](https://lightning.network/lightning-network.pdf" rel="noopener" target="blank" title=")

Furthermore, recall that once a block is mined, all other nodes must validate the block before accepting it. If the block size was incredibly large and somebody were to publish an invalid block, nodes would waste a large amount of time attempting to validate the block before discarding it as invalid and moving onto the next block. A denial of service attack can essentially be orchestrated by repeatedly publishing insanely large invalid blocks to the network, stopping valid blocks from being processed for a long period of time. As stated by blockchain pioneer Nick Szabo in this interview, the small block size acts as a technical security parameter to prevent network flooding.

You can read more about the full impact of an increased block size if Bitcoin were to take over the world, in an article I’ve written here.

If we can’t increase the block size, what can we do? Luckily, there are several solutions in the works that are expected to be deployed in order to solve this issue.

Segregated Witness (SegWit)

Segregated Witness (SegWit) has actually already been implemented into the Bitcoin network, as of August 2017. It’s a fundamental network change that modifies the format of transactions, essentially slimming them down in size, and allowing more transactions to be fit into a block which increases throughput. SegWit is considered a soft fork, meaning it is completely backwards compatible with existing Bitcoin protocol, although nodes and wallets must upgrade to take advantage of all SegWit features.

Each transaction has a signature from the sender, or in other words, witness data; this is usually the largest part of the transaction. This data is not actually necessary to verify the transaction, and so SegWit moves this data to the end of the transaction, segregating it. If this transaction is sent to a legacy node (a node that has not upgraded to SegWit), the node strips the witness data off the end of the transaction before inserting it into a block, thus reducing the overall transaction size and saving space. The added benefit of this is that nodes can no longer modify the witness data, changing who the transaction was from, an ability nodes previously had. This makes way for the implementation of multi-layer solutions that we’ll discuss soon. Users also save on transaction fees, as they’re usually calculated per transaction byte, and SegWit reduces total transaction size.

_SegWit moves signature data to the end of transaction, after which stripped before being stored in a block, source: [Programming Blockchains](https://programmingblockchain.gitbooks.io/programmingblockchain/content/other_types_of_ownership/p2wpkh_pay_to_witness_public_key_hash.html" rel="noopener" target="blank" title=")

Furthermore, SegWit changes the definition of a block: instead of a block being defined in terms of bytes, its now defined in terms of “weights”; a block can have a maximum weight of 4,000. Legacy transactions have a weight of 4, while SegWit transactions have a weight of 0.25, thus enabling a block to contain many more SegWit transactions and have a slightly higher size (approximately 2 megabytes at most). Nodes must upgrade to SegWit to follow this definition, and wallets must incorporate SegWit in order to send SegWit transactions. As a result, SegWit adoption has been slow, accounting for only 12% of current traffic.

_Current state of SegWit adoption hovers around 12% of all transactions, source: [segwit.party](http://segwit.party/charts/" rel="noopener" target="blank" title=")

Because of the aforementioned benefits of SegWit, I highly encourage anybody reading this to use wallets that have incorporated SegWit to speed up SegWit adoption. A neat list of them can be found here (my personal favourite is Samourai Wallet for Android). If you want to learn more about the intricacies of SegWit, Jimmy Song has written a great article about it:

Understanding Segwit Block Size
_After I wrote my last article, I was surprised by the protest about the 2MB part of the title (the title has since been…_medium.com

Multi-Layered Solutions

As it stands, the Bitcoin blockchain isn’t very feasible for micropayments. If you want to buy a $2 cup of coffee, you’re probably going to pay more than $2 equivalent of BTC in transaction fees, and the transaction won’t be confirmed instantly — you must wait for the transaction to be published in a verified block on the chain, which will appear within 10 minutes at best.

Second and third layer solutions are networks layered on top of the Bitcoin blockchain that enable users to send several transactions of small amounts of Bitcoin almost instantly, with no transaction fees.

The Lightning Network is this layered network currently in development expected to alleviate Bitcoin’s scaling problems. This network consists of two additional layers and enables users to open direct channels between each other to send an effectively unlimited number of payments to each other in an instant manner.

Lightning Network
_The Lightning Network is dependent upon the underlying technology of the blockchain. By using real Bitcoin/blockchain…_lightning.network

The Second Layer

A user joins the second layer network by conducting a transaction on the blockchain that declares the user is committing a certain number of bitcoin to be used in the layered network. The user then joins a group of nodes that are interconnected with one another, called channel factories. These nodes essentially uphold a lobby of individuals that potentially want to conduct transactions with one another. Channel factories then enable an unlimited number of micropayment channels to be created on the third layer (hence the name factories) between individual parties.

_From the [whitepaper](https://www.tik.ee.ethz.ch/file/a20a865ce40d40c8f942cf206a7cba96/Scalable_Funding_Of_Blockchain_Micropayment_Networks%20(1).pdf" rel="noopener" target="blank" title="): users are hooked into a channel factory upon joining the network, which then allocates multiple micropayment channels

The Third Layer

Micropayment channels are set up to ensure direct payments between two users on the third layer. Because the blockchain is no longer present in this layer, it cannot be used to validate transactions and ensure one party paid the other. Instead, smart-contract technology is employed, such as multisig addresses, meaning addresses that can be signed off by multiple users to enable the movement of funds, and hashed time-lock contracts, which are cryptographically secure automated contracts that lock funds for a certain period of time to ensure one parties that cannot cheat with another. These technologies eliminate the need for trust between users that are connected in micropayment channels.

Here is an example of how a Lightning Network micropayment channel works:

1. Alice wants to dedicate 1 Bitcoin to a micropayment channel between Bob. She declares this 1 Bitcoin to be used in a commitment transaction on the Bitcoin blockchain. This 1 Bitcoin is then locked up in a multisig address that both parties can sign off on if they want to close the channel. This address is secured with a hashed time-lock contract which states, “Alice has 1 BTC and Bob has 0 BTC, to be released in one hour”. This means the 1 Bitcoin Alice has is locked for 1 hour after which it will be returned to Alice and published to the Bitcoin blockchain once more.
2. Alice then decides to give Bob 0.1 BTC. This transaction is logged with a new hashed time-lock contract stating “Alice has 0.9 BTC and Bob has 0.1 BTC, to be expired in 50 minutes”. This contract has an expiry time of 50 minutes, meaning it will be published to the blockchain before the original contract stating Alice has 1 BTC. Therefore Bob instantly knows he has the 0.1 BTC because this new contract will be published to the blockchain before the original contract, essentially making the old contract invalid.
3. Once the full hour passes, the micropayment channel closes and the final balance between Alice and Bob is published to the blockchain. If Alice and Bob want to continue making transactions, they can extend the expiry time of their channel for as long as they want. If one of them wants to close the channel early, one of them needs to sign off on the original multisig address that the Bitcoin is stored in.

The network enables transactions to route itself to its final destination by using other connected users in the channel as intermediaries. This can happen even if a direct connection to the desired user isn’t able to be sought the current micropayment channel. For example, if Alice has a channel open with Bob, and Bob has a channel with Mark, and Alice wants to send Mark some Bitcoin, the network can route the payment to Mark through Bob, all while ensuring no party has to trust another.

In the lightning network, transactions are routed through intermediate users in order to reach its final destination

The implementation of lightning network transactions and their trust-less nature can get incredibly complex, and is best explained by the Lightning developers in this conference or in the following series by ecurrencyhodler:

The Lightning Network (Part 1)
_Multisig Addresses: The Building Blocks of the Lightning Network_medium.com

Ideally, a user will only create a commitment transaction to the secondary layer very rarely because he or she will remain in the layered network for prolonged periods of time to conduct most of their day-to-day transactions. Once a user wants to exit this multi-layered network, a settlement transaction is made on the blockchain declaring the user’s final Bitcoin balance after all of the second-layer activities. This reconciles their total Bitcoin balance on the blockchain after comparison with the original commitment transaction. In total, only two blockchain transactions are made in order to let the user to conduct a limitless number of transactions for free on the second layer.

As mentioned previously, SegWit paves the way for the lightning network because it removes nodes’ abilities to modify witness data, which is what is used to identify a user’s entry into the second layer. If the user’s commitment transaction can’t be found because the witness data referring to the user was changed, there is a greater level of difficulty involved when trying to reconcile the user’s settlement transaction.

The second layer of the lightning network involving channel factories was very recently introduced in this whitepaper. It is still under heavy development, so a lot of its concepts are explained abstractly. However, the network is poised to launch in 2018 and will be by far the biggest improvement in transaction scalability thus far.

Schnorr Signatures

When a user sends a Bitcoin transaction, the inputs of the transaction (the amount you’re sending) is calculated simply by retrieving from the blockchain the total unspent amounts of Bitcoin you previously received. So for example:

• Starting with an empty wallet, I receive 1 Bitcoin in transaction #1, and then another 1 Bitcoin in a separate transaction #2
• I now want to send 2 Bitcoins in a transaction. There will be two inputs to this transaction: transaction #1, and transaction #2, summing up to 2 Bitcoin

Under the current algorithm for generating signatures (Elliptic Curve Digital Signature Algorithm), each input requires its own signature. This increases the total transaction size and therefore increases the transaction fee.

Currently, each input requires a signature, increasing total transaction size

Schnorr signatures are an alternative and more efficient way of storing signature data in transactions. All inputs are accumulated and then stored as a single signature by utilizing the Schnorr algorithm, which greatly saves space in a transaction and further helps increase transaction throughput by allowing blocks to store more transactions on average.

All sender signatures are stored as one signature under the Schnorr algorithm

Schnorr signatures can be also be used to aid Bitcoin’s advancement in privacy by benefiting CoinJoin transactions. CoinJoin is a method of introducing anonymity to Bitcoin transactions. It works by pooling transaction inputs together with other peoples’ transactions when making a payment to a receiver. When payments are pooled, it becomes difficult to track which user sent what input, effectively making them anonymous. However, CoinJoin transactions have increased fees due to a higher number of inputs in a single transaction resulting in a higher number of signatures. Utilizing Schnorr signatures would enable all signatures in a transaction to be compressed into one, saving greatly on transaction fees and encouraging the use of CoinJoin.

Furthermore, Schnorr paves the way for complex multisig transactions which require signing off from multiple parties; no matter how many parties’ signatures are required for a transaction, all the transaction needs is one Schnorr signature.

Schnorr signatures are only now a possibility because of the implementation of SegWit; because signature data can’t be modified by third parties, it can now be used to effectively create a Schnorr signature.

MimbleWimble

MimbleWimble is a radical but incredibly powerful proposed change to Bitcoin architecture that was anonymously introduced through this whitepaper in July 2016.

Named after the tongue-tying curse from the Harry Potter series, its aim is to remove transactions entirely from blocks. Under MimbleWimble, transactions consist of nothing but input amounts, output amounts, and a signature. The signature of the transaction can only be decrypted by the receiver, and so transaction verification is left to the receiver.

By extension, blocks consist of only the list of all transaction input amounts of all transactions, all transaction output amounts, and their corresponding signatures. Blocks can then be merged seamlessly with previous blocks as they’re nothing but pairs of input and output amounts. Nodes then have the ability to cryptographically ensure that transactions in blocks do not create extra bitcoins (i.e. their net difference between inputs and outputs in blocks is 0) without having to decrypt transactions.

This removal of transaction storage grants complete anonymity to all users by stripping away the ability to generate transaction history. Furthermore, with blocks only containing the unspent transaction outputs (meaning the number of Bitcoins that have been received in an address but not moved out yet), the blockchain size can be reduced by over 60% according to the whitepaper. This reduction in size means that in order to validate a MimbleWimble blockchain, nodes will only need to look at the set of unspent transaction outputs instead of the entire set of transactions, which will exponentially increase performance.

The mathematical details of MimbleWimble are outside of the scope of this article, but are explained in detail in the whitepaper. Although MimbleWimble presents some clear advantages and technical breakthroughs, its implementation requires the removal of Bitcoin’s Script system that much of the existing architecture relies on. As a result, MimbleWimble’s implementation on the Bitcoin blockchain is not technically feasible.

However, there are proposals for MimbleWimble to exist as a sidechain. A sidechain is a separate blockchain directly attached to the Bitcoin blockchain through the use of a two-way peg. This peg enables assets between the two chains to be exchanged and “pegs” the value of the sidechain asset to the value of Bitcoin. In this setup, users would be able to exchange Bitcoins for MimbleWimble coins, conduct completely private and rapid transactions on the MimbleWimble chain, and then exchange their MimbleWimble coins for Bitcoin whenever they please.

_Sidechain coins are pegged to the Bitcoin blockchain, operating alongside it with a fixed exchange rate, source: [Blockchain.com](http://blockchain.com" rel="noopener" target="blank" title=")

In fact, a group of developers are already in the process of developing MimbleWimble as a separate cryptocurrency called GRIN; it was recently deployed on a test network and may be launched in the near future.

Rootstock

Rootstock — Smart Contracts on the Bitcoin Blockchain
_As a concept the Rootstock [1] platform is one of those ideas that once it is proposed it is obvious that it is a great…_medium.com

Rootstock is for whatever reason one of the less talked about advancements in Bitcoin technology but is by far one of the coolest. Rootstock is described as “the first open-source smart contract platform with a 2-way peg to Bitcoin that also rewards the Bitcoin miners via merge-mining, allowing them to actively participate in the Smart Contract revolution.”

Much like MimbleWimble, Rootstock is being developed as a sidechain solution to the Bitcoin blockchain. Its fundamental value lies in its focus in smart contracts. Rootstock aims to be a Turing Complete (completely programmable) smart-contracts platform that will be backwards compatible with Ethereum’s virtual machine. This means that Rootstock will be able to execute any smart contracts developed for the Ethereum platform and have smart contracts developed for its own platform.

Rootstock aims to implement this versatile smart-contract functionality all while leveraging Bitcoin’s comparatively dominant userbase and value by acting as a two-way pegged sidechain. It is also being designed to be secured by the existing Bitcoin mining network, therefore not needing to incentivize miners to secure its own blockchain. Rootstock also aims to tackle the transaction scalability problem by implementing its own version of a multi-layered solution called Lumino. With this, it may be able to accomplish up to 20,000 transactions per second.

Rootstock is aiming for a launch by the end of 2017. Overall, the platform aims to fit in perfectly alongside Bitcoin and if its claims hold true, it will undoubtedly bring unprecedented utility to the Bitcoin network.

If you’ve gotten this far, congratulations! I hope you were able to learn something about the future of Bitcoin and are as excited about it as I am.

Bitcoin isn’t perfect and is faced with challenges that its community must work to solve. However, it is backed by an incredibly dedicated and thriving developer community that is working day in and day out to tackle these problems. There are constant innovations happening everyday, and I am sure by the time you finished reading this another new and exciting proposal for the Bitcoin blockchain popped up.

This article by no means covers everything out there; if there’s anything that you’re aware of that I didn’t mention, please mention them in the comments!

Follow me on Twitter and Medium if you’re interested in more in-depth and informative write-ups like these in the future!

I’m a relative Bitcoin beginner myself so if there are any mistakes or if you have any feedback please don’t hesitate let me know!

BTC Address: 3MGguJhw1bm95tDQjZ3b8ySBwZLJ77CgG1

Here are some resources if you’re interested in learning more:

• Mastering Bitcoin, an comprehensive book written by Andreas M. Antonopoulos
• The Bitcoin Wiki
• The Bitcoin whitepaper
• A collection of Bitcoin resources by Jameson Lopp
• This highly informative interview with Nick Szabos conducted by Tim Ferriss
• Ivan on Tech on Youtube, who is absolutely amazing at breaking down technical Bitcoin concepts