I previously wrote an article showing how to create a Figma/Miro Clone in React and TypeScript. The code in the article was designed to be as easy to understand, and in this article, we’re going to optimize it. The code used DndKit for dragging and dropping, and D3 Zoom for panning and zooming. There were four components (App, Canvas, Draggable and Addable), and about 250 lines of code. You do not need to read the original article to understand this one.

Standard optimizations such as useCallback, memo, and similar made it about twice as fast when dragging, but made no difference for panning and zooming. More creative/intensive optimizations made it about ten times as fast in most cases.

You can see the optimized code on GitHub and there is a live demo on GitHub pages to test out the speed with 100,000 cards.

Table of Contents

• How to Measure Performance in React Apps

• How to Investigate the performance

• How to Optimize Panning and Zooming the Canvas

• How to Optimize Dragging Cards Around the Canvas

• Results

• Summary

How to Measure Performance in React Apps

There are three common ways to measure performance in React Apps

• React Dev Tools profiler

• Chrome Dev Tools profiler, especially using custom tracks

• Profiler component

These tools are all great, but none of them are quite the right fit in this case. In most codebases, the time spent executing JavaScript code (both our code and that of the React framework) is the primary issue. However, after all your code has run and React has updated the Dom, the browser still has a lot of work to do:

In this case, this browser layout and rendering time was significant, and is not accounted for by the React profiling.

You can use custom tracks in the Chrome dev tools profiler, but it is very cumbersome to use.

For us, the JavaScript performance API is the best option, which gives results that are closer to those experienced by the user, and is relatively easy to use.

First, we make a call to performance.mark in the event handler that starts the action, with a string to describe the time point. For example, when starting a zoom or pan operation:

zoomBehavior.on("start", () => {
    performance.mark('zoomingOrPanningStart');
}

Then, in a useEffect hook, we call performance.mark again, and call performance.measure to calculate the time between the two points:

useEffect(() => {
    performance.mark('zoomingOrPanningEnd');
    performance.measure('zoomingOrPanning', 'zoomingOrPanningStart', 'zoomingOrPanningEnd');
});

The React docs states that useEffect usually fires after the browser has painted the updated screen, which is what we want.

This isn't perfect, and will vary depending on the machine specifications, and what else the machine is doing at the time, but it was good enough to verify which optimizations worked best. It is possible to go further if you need to. For example, using Cypress to automate and profile scenarios, potentially running many times to get a good mean, or using Browserstack to test on a variety of devices.

How to Investigate the Performance

Most of the investigation involved using the React Dev Tools profiler to record profiles of user interactions.

The performance data shows how many commits there were in the profile, and how long each one took, which is a great way to see if there are too many commits.

Each commit displays a flame chart showing which components rendered and why they re-rendered. This makes it much easier to find ways to avoid the re-rendering, and to check that memoization strategies are working as expected. This does have some caveats though. It often says 'The parent component rendered', which is misleading default text for when it doesn’t understand what happened (and is often due to a change in a parent context). It also says things like 'hook 9 changed', which makes it time consuming to work out exactly which hook changed.

The flame chart also shows how long each component took to render. This helps target problem components that we need to focus on.

How to Optimize Panning and Zooming the Canvas

The original Canvas element used the CSS transform translate3d(x, y, k) to pan and zoom the canvas. This works, but it doesn't scale child elements, so when the zoom changes, all the cards on the canvas have to be re-rendered with a new CSS transform for the new zoom level (scale(${canvasTransform.k})).

 <div
    ...
    className="canvas"
    style={{
        transform: `translate3d(${transform.x}px, ${transform.y}px, ${transform.k}px)`,
        ...
    }}>
    ...
</div>
<div
    className="card"
    style={{
        ...
        transform: `scale(${canvasTransform.k})`,
    }}>
    ...
</div>

I changed this to use translateX(x) translateY(y) scale(k), which has the same effect, but does scale child elements. This way, when the zoom changes, none of the cards will be re-rendered (the style of the card component no longer uses the canvasTransform.k).

 <div
    ...
    className="canvas"
    style={{
        transform: `translateX(${transform.x}px) translateY(${transform.y}px) scale(${transform.k})`,
        ...
    }}>
    ...
</div>
<div
    className="card"
    ...
</div>

The Canvas still needed to re-render whenever the pan or zoom changed, and it is possible to prevent this with useRef, and updating the CSS transform with direct JavaScript Dom manipulation in the d3-zoom event handler. This doesn’t make a significant improvement to the performance though, and is a definite hack, so the trade off is not worthwhile.

Both zooming and panning get a bit slower when the canvas is zoomed very far out and there are (a lot) more cards visible on the screen, just due to the browser having to render them all. It's still workable at 100,000 cards though. There are things you can do about this. An easy option is limiting the maximum zoom extent. This is a functional change, so potentially something that doesn’t meet requirements, but it is easy to do in d3-zoom using scaleExtent:

zoom<HTMLDivElement>().scaleExtent([0.1, 100])

Another option is to create a bitmap for very low zoom levels and render that as a single element. This may be difficult, but it means that there will be no change to the functionality.

How to Optimize Dragging Cards Around the Canvas

Starting a drag

The useDraggable hook from DndContext causes some re-renders when starting a drag operation.

It is possible to improve this by changing the Draggable component to just have this hook (and the things that use it) and having a DraggableInner component for everything else (inside a memo). This works well for reducing the re-renders, in that the DraggableInner almost never get re-rendered, and improves the speed of starting a drag operation. However, it was still fairly slow, and the time was all under the DndContext.

A better option is to create a new NonDraggable component, that looks exactly like the Draggable component, but does not hook up with DndContext. These cards are shown on the Canvas, and have an onMouseEnter event, to swap in the Draggable component for the active card, so that dragging continued to work.

const onMouseEnter = useCallback(() => {
    setHoverCard(card);
}, []);

This works well, and significantly improves the speed when starting a drag operation, but it was still quite slow with large numbers of cards. Nearly nothing was getting re-rendered, but there is still a time cost to when using memo, as it needs to check whether components have changed.

To fix this, we create an AllCards component, that contains all the cards on the canvas as NonDraggable components. Because it always renders all the cards, it nearly never needs to be re-rendered, and it is used with memo. So instead of each individual card using a memo (with the associated time cost), there is now just one component using a memo. To make it so that the dragging still works, the active Draggable component is rendered on top, obscuring the NonDraggable component beneath it. There is also a Cover component beneath that, so that when the Draggable component is dragged away, the NonDraggable component underneath remains hidden.

Original code, where each card is a Draggable component:

<DndContext ...>
    {cards.map((card) => (
        <Draggable card={card} key={card.id} canvasTransform={transform} />
    ))}
</DndContext>

Optimized code, where the AllCards component renders all the cards as NonDraggable components, and then a Cover and a Draggable component for the active card.

<AllCards cards={cards} setHoverCard={setHoverCard} />
<DndContext ...>
    <Cover card={hoverCard} />
    <Draggable card={hoverCard} canvasTransform={transform} />
</DndContext>

This works very well. With a low number of cards, the speed is about the same, but with a high numbers of cards, it’s about twenty times faster.

There is now a new potential performance issue with the onMouseEnter event that swaps in the Draggable component for the active card, but this just adds two components to the Dom, and is very quick even with large numbers of cards.

Finishing a drag

Finishing a drag operation is hard to optimize, as the position of a card changes, and that does need to re-render, which means that the AllCards component has to re-render as well.

You can see original code below. Even when using memo with the Draggable component, the end drag operation still takes 2500ms with 100,000 cards, mostly due to the complexity of the Draggable component and its integration with DndKit.

<DndContext ...>
    {cards.map((card) => (
        <Draggable card={card} key={card.id} canvasTransform={transform} />
    ))}
</DndContext>

However, we now use the NonDraggable components, which all memo successfully, and only the dragged card is re-rendered. There is still a time cost using the memo, and this is the slowest part of the solution, but it leads to an increase in speed to 500ms with 100,000 cards.

const NonDraggable = memo(...)

const AllCards = memo((cards, setHoverCard) => {
    <>
        {cards.map((card) => {
            <NonDraggable card={card} key={card.id} setHoverCard={setHoverCard} />);
        })}
    </>;
});

Results

The base unoptimized version started to get slow between 1000 and 5000 cards. Standard optimizations improved this to around 10,000 cards, and the more optimization took it to about 100,000 cards. The trade off is that the code becomes significantly more complicated, which makes it harder to understand and modify, especially for people new to the codebase.

Summary

It is unusual to display 100,000 or more items on screen in a standard React App, but in a highly graphical codebase, it becomes much more likely.

With these numbers, the browser rendering engine is likely to take a significant amount of time, so it is best to use the performance API to measure performance, instead of the usual React tools.

Standard React optimization strategies do work and improve the situation, but there is a need to go further, by finding ways to avoid renders, and even to avoid too many memo comparisons.

Instead of sticking post it notes on a physical wall, you can now add virtual post its (and a dizzying array of other things) to a virtual canvas. This lets teams collaborate virtually in ways that feel familiar from the physical world.

Figma and Miro are large, mature products, and they bypass HTML and CSS entirely. They use technologies like WebAssembly, WebGL, C++ and similar, due to their extremely demanding performance requirements.

But we can create similar virtual canvas type features, without the complexity, using React, TypeScript, and a couple of packages. We are going to support one type of 'card', that will just contain simple text, to keep this guide concise, but it is easy to extend the solution to support more elaborate use cases.

The functionality we are going to implement is:

• Dragging cards around the canvas

• Adding new cards to the canvas

• Panning and Zooming the canvas

This article is a step by step guide describing how to create these features, and there is companion code on GitHub with live demos.

Our scratch-built solution won't be as fast as Figma or Miro, but if you're needs are simpler, it'll probably be enough.

Project Overview

We'll use DndKit for dragging and dropping and D3 Zoom for panning and zooming. I found both of these tools to be a pleasure to work with.

The code is on GitHub, and there is a live demo so you can try it out.

There are only 4 components (App, Canvas, Draggable and Addable), and only about 250 lines of code.

This guide is aimed at intermediate level React / TypeScript developers. It will be much easier if you already understand components, hooks, how to think in React, state, mutable refs and things like that.

Screen grab showing final canvas solution

Alright, now to see how you can actually build this, let's dive in.

Step 1 – How to Drag the Cards Around the Canvas

To get started, we'll use DndKit to drag cards around a canvas. We'll install the tools we need to build the project, create a simple Card type, and create simple App, Canvas and Draggable components.

The App component stores the card state and renders the Canvas.

The Canvas component integrates with DndKit, updates the card state and renders the cards as Draggable components.

The Draggable component integrates with DndKit and uses CSS styling to position itself correctly on the canvas.

Step 1 Code on GitHub.

Here's a screen grab of what we'll be building in this part, and there is also a live demo you can try out for yourself:

Screen grab showing part 1 of the canvas solution

Project Setup

We create a new project with Vite and install DndKit using the following commands:

npm create vite@latest figma-miro-canvas -- --template react-ts
npm install
npm install @dnd-kit/core

App.tsx

The App component stores the card state and renders the Canvas.

We add a Card type, in this demo cards will simply contain some text. UniqueIdentifier is from DndKit, and looks scary, but it's just a String | number. Coordinates also comes from DndKit and contains the x and y position.

export type Card = {  
  id: UniqueIdentifier;
  coordinates: Coordinates;
  text: string;
};

We then need to create some cards, and pass them to the canvas.

export const App = () => {
  const [cards, setCards] = useState<Card[]>([
    { id: "Hello", coordinates: { x: 0, y: 0 }, text: "Hello" },
  ]);

  return (<Canvas cards={cards} />);
}

App.tsx on GitHub

Canvas.tsx

The Canvas component integrates with DndKit, updates the card state and renders the cards as Draggable components.

It takes cards and setCards as props, as the state is stored higher up the tree in App. This isn't strictly necessary right now, but is useful in later steps.

type Props = {
  cards: Card[];
  setCards: (cards: Card[]) => void;
}

We add a function to call setCards with an update after a drag operation has completed. This simply adds the drag distance / delta to the x and y values for the card that was being dragged.

The DragEndEvent comes from DndKit, and includes the active item being dragged, so we can use that to work out which card to update.

const updateDraggedCardPosition = ({ delta, active }: DragEndEvent) => {
  if (!delta.x && !delta.y) return;

  setCards(
    cards.map((card) => {
      if (card.id === active.id) {
        return {
          ...card,
          coordinates: {
            x: card.coordinates.x + delta.x,
            y: card.coordinates.y + delta.y,
          },
        };
      }
      return card;
    })
  );
};

We add a div to represent the canvas, a DndContext (from DndKit), and render a Draggable for each card. We hook up our new function with the onDragEnd event from DndContext so that the card state is updated after a successful drag operation.

<div className="canvas">
  <DndContext onDragEnd={updateDraggedCardPosition}>
    {cards.map((card) => (
      <Draggable card={card} key={card.id} />
    ))}
  </DndContext>
</div>

Canvas.tsx on GitHub

Draggable.tsx

The Draggable component integrates with DndKit and uses CSS styling (position, top and left) to position itself correctly on the canvas.

useDraggable comes from DndKit, and we blindly pass the attributes, listeners, and setNodeRef that it returns to our div, which allows it to respond to onClick events and things like that.

We also use the transform from DndKit to apply CSS in order to render the card at a modified position when a drag is in progress. Slightly confusingly, the CSS property name is also called transform.

export const Draggable = ({ card }: { card: Card }) => {
  // hook up to DndKit
  const { attributes, listeners, setNodeRef, transform } = useDraggable({
    id: card.id,
  });

  return (
    <div
      className="card"
      style={{
        // position card on canvas
        position: "absolute",
        top: `${card.coordinates.y}px`,
        left: `${card.coordinates.x}px`,
        // temporary change to this position when dragging
        ...(transform
          ? {
              transform: `translate3d(${transform.x}px, ${transform.y}px, 0px)`,
            }
          : {}),
      }}
      ref={setNodeRef}
      {...listeners}
      {...attributes}
    >
      {card.text}
    </div>
  );
};

Draggable.tsx on GitHub

App.css

Finally we can add some CSS styling to make everything look vaguely acceptable. This isn't strictly necessary, but it does look a lot better, even with my limited design skills.

App.css on GitHub

Step 2 – How to Add New Cards to the Canvas

In this step, we'll create a new Addable component, to represent cards that are not currently on the canvas, but can be dragged on to it.

We will update App to add a "tray" div to contain these new Addable cards. We will also add another DndContext (this new DndContext will handle the drag drop from the tray to the canvas, and the existing DndContext in Canvas handles dragging of cards around the canvas), and hook up to its events. This will let us update the state when the Addable cards are drag / dropped on to the canvas.

We will update Canvas to make it a DndKit drop target, so that the Addable cards can be dropped on to it.

Step 2 code on GitHub

Here's the functionality we'll add in this section, and there is also a live demo you can try out for yourself:

App.tsx

The App component now needs a "tray" div to contain Addable cards.

It also needs another DndContext (this new DndContext will handle the drag drop from the tray to the canvas, and the existing DndContext in Canvas handles dragging of cards around the canvas). It needs to hook up to its events so that we can update the state when the Addable cards are drag / dropped on to the canvas.

We add a list of cards to appear on the tray:

const trayCards = [
  // the coordinates aren't used for the tray cards, we could create a new type without them
  { id: "World", coordinates: { x: 0, y: 0 }, text: "World" },
];

We add a function to work out the position on the canvas at the end of a drag drop operation. This has to know the initial position of the card, the drag distance / delta, and the top left position of the canvas. These details are all provided by the DndKit DragEndEvent.

const calculateCanvasPosition = (
  initialRect: ClientRect,
  over: Over,
  delta: Translate
): Coordinates => ({
  x: initialRect.left + delta.x - (over?.rect?.left ?? 0),
  y: initialRect.top + delta.y - (over?.rect?.top ?? 0),
});

We add state to store the tray card being dragged, along with a function to update the cards state after a drag / drop from the tray. The DragEndEvent comes from DndKit, and includes the active item being dragged, so we can use that to create a new card and add it to the cards array. We also make some checks to make sure that the "canvas" is the drop target, and that we have all the data we need.

const [draggedTrayCardId, setDraggedTrayCardId] = useState<UniqueIdentifier | null>(null);

const addDraggedTrayCardToCanvas = ({ over, active, delta }: DragEndEvent) => {
  setDraggedTrayCardId(null);

  if (over?.id !== "canvas") return;
  if (!active.rect.current.initial) return;

  setCards([
    ...cards,
    {
      id: active.id,
      coordinates: calculateCanvasPosition(
        active.rect.current.initial,
        over,
        delta
      ),
      text: active.id.toString(),
    },
  ]);
};

We add the new, additional, DndContext, a div to represent the tray, and a DragOverlay.

The DragOverlay component comes from DndKit, and we render the tray card being dragged inside it. It does the hard work of showing the tray card while it is being dragged. It is very convenient to use with Drag / Drop, but not as handy when just dragging, which is why we didn't use one earlier when dragging cards around the canvas.

<DndContext
  onDragStart={({ active }) => setDraggedTrayCardId(active.id)} 
  onDragEnd={addDraggedTrayCardToCanvas} 
>
  <div className="tray">
    {trayCards.map((trayCard) => {
      // this line removes the card from the tray if it's on the canvas
      if (cards.find((card) => card.id === trayCard.id)) return null;

      return <Addable card={trayCard} key={trayCard.id} />;
    })}
  </div>
  <Canvas cards={cards} setCards={setCards} />
  <DragOverlay>
    {/* this works because the id of the card is the same as the text in this example so we can just render the id inside a div. In more complex cases you would have a component to render the card, and use that here. */}
    <div className="trayOverlayCard">{draggedTrayCardId}</div>
  </DragOverlay>
</DndContext>

App.tsx on GitHub

Addable.tsx

The Addable component integrates with DndKit, and is used to represent cards that are not currently on the canvas, but can be dragged on to it.

We hook up the component with DndKit and render the card text in a div. useDraggable comes from DndKit, and we blindly pass the attributes, listeners and setNodeRef that it returns on to our div. This allows it to respond to onClick events and things like that.

export const Addable = ({
   card
} : {
  card: Card;
}) => {
  const { attributes, listeners, setNodeRef } = useDraggable({
    card.id,
  });

  return (
    <div 
      className="trayCard"
      ref={setNodeRef}
      {...listeners}
      {...attributes}>
      {card.text}
    </div>
  );
};

Addable.tsx on GitHub

Canvas.tsx

Canvas now needs to integrate with DndKit to make it a drop target, so that the Addable cards can be dropped on to it.

We hook up the canvas as a drop target with DndKit. useDroppable comes from DndKit, and we just pass the ref on to our div. This is so that DndKit can identify it, and obtain its id when it is dragged over.

const { setNodeRef } = useDroppable({ id: "canvas" });

<div
  className="canvas"
  ref={setNodeRef}
  ...
>
  ...
</div>

Canvas.tsx on GitHub

Step 3 - How to Pan Around and Zoom In and Out from the Canvas

In this final step, we'll install d3-zoom, hook it up to the canvas, and then update some calculations and styles so that everything appears in the right place.

We will update App, to store the transform (the pan and zoom of the canvas) from d3-zoom, and update the style of the DragOverlay and calculateCanvasPosition to take account of the transform.

We will update Canvas to integrate with d3-zoom and to use CSS styling to take account of the transform.

We will update Draggable, using CSS styling to take account of the transform, both when stationary and whilst being dragged.

d3-zoom handles mouse and pointer events for pan and zoom automatically, so we don't need to add any code for that (but it's easy to do so if you want to add a "Zoom In" button or similar).

Step 3 code on GitHub

Here's what we'll build in this section, and there is also a live demo you can try out for yourself:

Before you continue on, make sure you've installed d3-zoom:

npm install d3-zoom
npm install --save-dev @types/d3-zoom

App.tsx

App now needs to store the transform (the pan and zoom of the canvas) from d3-zoom and update the style of the DragOverlay and calculateCanvasPosition to take account of the transform.

We store the current transform from d3-zoom. This represents both the pan (transform.x and transform.y) and the zoom (transform.k).

const [transform, setTransform] = useState(zoomIdentity);

We add CSS to the DragOverlay, so that cards dragged from the tray are the same size that they are on the canvas.

style={{
  transformOrigin: "top left",
  transform: `scale(${transform.k})`,
}}

We update the calculateCanvasPosition function as it now needs to account for the zoom of the canvas (transform.k) as well as the top left position.

const calculateCanvasPosition = (
  initialRect: ClientRect,
  over: Over,
  delta: Translate,
  transform: ZoomTransform
): Coordinates => ({
  x: (initialRect.left + delta.x - (over?.rect?.left ?? 0) - transform.x) / transform.k,
  y: (initialRect.top + delta.y - (over?.rect?.top ?? 0) - transform.y) / transform.k,
});

App.tsx on GitHub

Canvas.tsx

Canvas now needs to integrate with d3-zoom and to use CSS styling to take account of the transform from d3-zoom.

We add props for transform and setTransform (we pass these down from App.tsx).

type Props = {
  cards: Card[];
  setCards: (cards: Card[]) => void;
  transform: ZoomTransform;
  setTransform(transform: ZoomTransform): void;
}

We hook up d3-zoom. Both DndKit and d3-zoom need a ref to the element, so we create canvasRef and updateAndForwardRef, which allows both of them to reference the same HTMLDivElement.

d3-zoom is a JavaScript library, rather than a React component, which is why we have to use the slightly esoteric code below, such as useMemo and useLayoutEffect (although you will see both of these in nearly any reasonably sized React codebase).

const canvasRef = useRef<HTMLDivElement | null>(null);

const updateAndForwardRef = (div: HTMLDivElement) => {
  canvasRef.current = div;
  setNodeRef(div);
};

// create the d3 zoom object, and useMemo to retain it for rerenders
const zoomBehavior = useMemo(() => zoom<HTMLDivElement, unknown>(), []);

// update the transform when d3 zoom notifies of a change.
const updateTransform = useCallback(
  ({ transform }: { transform: ZoomTransform }) => {
    setTransform(transform);
  },
  [setTransform]
);

useLayoutEffect(() => {
  if (!canvasRef.current) return;

  // get transform change notifications from d3 zoom
  zoomBehavior.on("zoom", updateTransform);

  // attach d3 zoom to the canvas div element, which will handle
  // mousewheel, gesture and drag events automatically for pan / zoom
  select<HTMLDivElement, unknown>(canvasRef.current).call(zoomBehavior);
  },
  [zoomBehavior, canvasRef, updateTransform]
 );

We add a wrapper / window around the canvas. The canvas div will now pan and zoom (so will move around the screen a lot), so we wrap it in another div with a fixed position and size and hide any overflow, so that we have a "window" showing the relevant part of the canvas.

We also add CSS styles to the canvas to account for the pan and zoom, use the new updateAndForwardRef function and move the ref from the canvas to the canvas window.

<div ref={updateAndForwardRef} className="canvasWindow">
  <div
    className="canvas"
    style={{
      // apply the transform from d3
      transformOrigin: "top left",
      transform: `translate3d(${transform.x}px, ${transform.y}px, ${transform.k}px)`,
      position: "relative",
      height: "300px",
    }}
  >
    ...
  </div>
</div>

Canvas.tsx on GitHub

Draggable.tsx

Draggable now needs different CSS styling to take account of the d3-zoom transform, both when stationary and whilst being dragged.

We add a prop for the d3-zoom transform, which we call canvasTransform, as we are already using a transform variable from DndKit.

type Props = {
  card: Card;
  canvasTransform: ZoomTransform;
}

We update the CSS to account for the canvas pan and zoom. We have to handle two cases, both when it is currently being dragged, and when it is not.

style={{
  position: "absolute",
  top: `${card.coordinates.y * canvasTransform.k}px`,
  left: `${card.coordinates.x * canvasTransform.k}px`,
  transformOrigin: "top left",
  ...(transform
    ? {
        // temporary change to this position when dragging
        transform: `translate3d(${transform.x}px, ${transform.y}px, 0px) scale(${canvasTransform.k})`,
      }
    : {
        // zoom to canvas zoom
        transform: `scale(${canvasTransform.k})`,
      }),
}}

We also stop the onPointerDown event bubbling up to the canvas, otherwise it would be handled by d3-zoom, and interpreted as a request to start dragging, which results in dragging the canvas and the card at the same time (an interesting but undesirable effect!)

onPointerDown={(e) => {
  listeners?.onPointerDown?.(e);
  e.preventDefault();
}}

Draggable.tsx on GitHub

Conclusion

There is some complexity to the various position / transform calculations, but it isn't too crazy, and there are only two dependencies to install.

There are only four components (App, Canvas, Draggable and Addable), and only about 250 lines of code for all of them, which seems like a very modest amount for all the functionality.

This demo is very simple, but it contains a lot of virtual canvas functionality, and it is easy to use this as a base and build something more elaborate on top.

So here, hopefully, is a good one – with a non trivial and real life example, what changes to support, and a description of the trade offs.

The Open-Closed principle states that code should be "Open for extension" and "Closed for modification".

There is quite a lot of confusion about the term, but essentially it means that if you want to implement a supported change, you should be able to do it without changing the code in a large number of places. Ideally, you can implement the new feature just by adding new code, and changing little or no old code, which makes the code easier to develop and maintain.

Open is the 'O' in the SOLID design principles, which are probably the most famous guides for writing high quality code.

No useful code can ever be completely open to all possible changes, so we have to decide which changes we are going to support. When writing our code we can think about what the potential changes might be, decide which ones to support, and then make the code 'open' to these.

We can create a list of these potential changes by:

• Analysing the code
• Looking at previous changes to the code
• Using our experience of commonly requested changes
• Using any knowledge of upcoming feature requests

Take a minute to look at the code below (and on GitHub) and think about what changes we might expect. You don't have access to the commit history, or any knowledge of upcoming feature requests, but you can still probably come up with some likely candidates.

public class GrossToNetCalculator
{
    public GrossToNetCalculator(
        IGrossEnergyYield grossYield,
        double grossEnergy,
        double hysteresisLoss,
        double curtailmentLossGrid,
        double turbineLossTurbulence,
        double electricalLoss,
        double turbineLossShear,
        double turbinePerformanceExperience,
        double operationalExperienceLoss)
    {
        double dependentLoss = 
            CombinePercentages(
                grossYield.TurbineAvailability,
                grossYield.BalanceAvailability,
                grossYield.AccessibilityAvailability,
                hysteresisLoss,
                electricalLoss,
                grossYield.EnvironmentalShutdownWeather,
                grossYield.EnvironmentalSiteAccess,
                grossYield.EnvironmentTreeGrowth);

        double independentLoss = 
            CombinePercentages(
                grossYield.GridAvailability,
                turbinePerformanceExperience,
                turbineLossTurbulence,
                grossYield.EnvironmentalPerformanceDegradationIcing,
                grossYield.CurtailmentPowerPurchase,
                grossYield.SubOptimalPerformance,
                turbineLossShear,
                operationalExperienceLoss);

        GrossToNet = 
            1 - 
            (1 - (dependentLoss + curtailmentLossGrid))
            * (1 - independentLoss);
    }

    double CombinePercentages(params double[] percentages)
    {
        double combination = 1;
        foreach (var percentage in percentages)
            combination *= 1 - percentage;
        return 1 - combination;
    }

    public double GrossToNet { get; private set; }
}

This code is relatively simple, and when I look at it these are the potential changes that I see:

• Items could be added or removed from the dependentLoss and independentLoss calculations. Items could be also be moved between dependentLoss and independentLoss, but this is essentially the same thing
• The calculation of GrossToNet could change
• The CombinePercentages calculation could change

As with most things in computer programming, there is a tension when applying the Open-Closed Principle.

On the one hand, making the code more easily extensible is good. On the other hand, doing this often breaks encapsulation, adds complication, and adds unnecessary levels of abstraction.

So again, we need to make a decision about which of these changes we want to support and make the code 'open to'. We can then avoid adding unnecessary complication to the code for unsuitable changes.

It is worth remembering that the work can always be done later, when it will be easier, as we will know exactly what is required.

To make a decision about what changes we should support and make the code 'open to', we need to estimate how likely the change is to occur, think about design solutions, and then think about the trade offs.

We Could Add or Remove Items from the dependentLoss and independentLoss Calculations

Very likely to change

The calculation of dependentLoss and independentLoss (for example double dependentLoss = CombinePercentages(...)) each use 8 parameters (electricalLoss, TurbineAvailability and so on).

These 16 make up the majority of the 17 total inputs to the entire calculation. So, from a purely statistical point of view, a change to one of these has a 16/17 (94%) chance of affecting these calculations.

It's also easy to imagine that we might want to add another "Loss" or "Availability" or similar in the future, or that a current one is no longer relevant, or that different combinations will be required in different circumstances.

Possible solution

Take a list of dependent and independent losses in the constructor, instead of taking each loss individually. So the existing constructor:

public GrossToNetCalculator(
    ...
    double hysteresisLoss,
    double curtailmentLossGrid,
    ...)

is replaced with this:

public GrossToNetCalculator(
    IReadOnlyList<double> dependentLosses
    IReadOnlyList<double> independentLosses)

This means that if the change is requested, we can implement it without changing the class (and instead just changing the parameters we pass to the constructor).

For example, if another 'dependentLoss' is requested, we can just add this to the dependentLosses list.

(You can see the code on GitHub here)

Trade offs

A small amount of encapsulation is lost, and the calling code would now be in charge of deciding which losses to pass in.

The code adheres much better to the Open-Closed Principle and becomes much more easily extendable and reusable. If you need to make a change, you won't need to modify the tests, which is useful as they are complicated.

Tests for the calling code would have to modified, but only to verify that they pass the correct parameters, which is much simpler.

It is possible that the constructor parameters are passed around in the code base, and now there are only two parameters, as opposed to the previous nine.

Decision

We should implement this solution to support this change and make the code 'open' to it.

The GrossToNet Calculation Could Change

Unlikely to change

The GrossToNet calculation is GrossToNet = 1 - (1 - (dependentLoss + curtailmentLossGrid)) * (1 - independentLoss);

Only the curtailmentLossGrid parameter is used, aside from the dependentLoss and independentLoss which are covered earlier.

This 1 parameter is a small minority of the 17 total inputs to the entire calculation. So, from a purely statistical point of view, a change to one of these has a 1/17 (6%) chance of affecting this calculation.

Possible solutions

1. Take a lambda parameter in the constructor to calculate GrossToNet and pass it dependentLoss and independentLoss, so that the calculation becomes GrossToNet = grossToNetCalculatorLambda(dependentLoss, independentLoss)(code on GitHub)
2. Remove curtailmentLossGrid from the calculation, which then becomes completely generic and can be renamed to PercentageCombiner. Require that the calling code applies this adjustment (this adjustment is too complicated for useful example code)
3. Remove curtailmentLossGrid from the calculation as above, then recreate the original GrossToNetCalculator, using the PercentageCombiner and adding curtailmentLossGrid to the calculation
   (code on GitHub)

Trade Offs

A large amount of encapsulation is lost for options 1 and 2. Option 3 is a reasonable amount of work, and adds a layer of abstraction.

Decision

This change isn't likely to happen, so it probably isn't worth the effort involved to support it and make the code 'open' to it. But if we had another use for the new PercentageCombiner then it would definitely be worthwhile.

The CombinePercentages Calculation Could Change

Very unlikely to change

CombinePercentages(params double[] percentages)
{
    double combination = 1;
    foreach (var percentage in percentages)
    combination *= 1 - percentage;
    return 1 - combination;
}

The CombinePercentages calculation implements some standard laws of math / statistics, which do not change.

Possible solutions

1. Take a lambda parameter in the constructor to combine the percentages, and use this instead of the CombinePercentages function. So instead of having double dependentLoss = CombinePercentages(...), you would have double dependentLoss = combinePercentagesLambda(...).
   (code on GitHub)
2. Create a PercentageCombiner abstraction, take this in the constructor to combine the percentages, and use this instead of the CombinePercentages function. So instead of having double dependentLoss = CombinePercentages(...), you would have double dependentLoss = percentageCombiner.CombinePercentages(...).
   (code on GitHub)

Trade offs

Combining percentages is at the heart of what this code does, so removing this logic makes the code mostly useless.

Option 1 passes all the responsibility for this on to the caller, whereas option 2 at least allows for predefined implementations of the abstraction.

Decision

This change is very unlikely, and the only reasonable solution (option 2) requires a lot of work and adds complexity and abstraction.

This means that it would only make sense to do it when the change is actually required, and even then only if multiple algorithms are required. Note that if a change is required to the algorithm, it will make more sense to simply change the implementation of the CombinePercentages function.

Conclusion

Deciding whether code adheres to the Open-Closed Principle is almost always a judgement call, and there are usually trade offs involved with encapsulation, complexity and abstraction.

It is worth thinking about likely changes and extensions, and using these to guide your decisions.

For example, if a test looks like this, it is hard to know what the problem is when it fails, as the test data is all hard coded. The test tells you very little about how the system under test (SUT) should behave, and so doesn’t act as documentation.

assert sut.wake_erosion_rate(0.03) == 0.8

However, if you remove this code smell, and include the calculation in the test, it becomes obvious what the problem is when there is a failure, and the test does now act as documentation.

You might also want to replace the 2.5 and 0.05 with the names of the constants they represent.

ambient_turbuluence = 0.03

assert 
    sut.wake_erosion_rate(ambient_turbuluence) 
    == 2.5 * ambient_turbuluence + 0.05

This article references the excellent XUnit Test Patterns book, which defines the most widely accepted lexicon on unit test patterns and practices.

Example Calculation

The rest of this article will discuss the example ConstructionMarginCalculator class, and will describe refactorings that simplify the tests and allow them to feasibly include the calculations.

The calculation is around 30 lines long, and is worth a quick glance now if you have the time. But if not, don’t worry, the rest of this post will still make sense.

There are a few if statements and a loop, and in total these lead to 2^9, or 512, paths through the code! Eek! It is clearly not feasible (or useful) to test all these, hence the need to find ways to make it easier.

This example code has an initial naive test, which is around 30 lines long. It doesn’t include the calculations, and introducing them would make the test even longer and more complicated, so would be hard to justify.

Each of the following sections describes a refactoring, and includes links to the refactored example code. These build on the existing test refactorings from XUnit Test Patterns.

The refactorings reduce the number of paths through the code and simplify the test. This means that fewer tests are needed, and that the tests become small and simple enough to include the calculations.

Extract Calculation From Loop

The code initially calculates values for a list of steps, which means that any test for it must take on this complication.

The setup of the test is harder, as we have to create a list as opposed to a single value. The assertions are harder as we have to assert on a list as opposed to a single value. Finally, we should have multiple tests (probably for 0, 1, and multiple items in the list).

The easy fix is simply to refactor the code so that it only calculates a single step. This displaces the iteration code to some other place, which might want testing. However, this can be tested using a Mock, so only the iteration needs to be tested (as opposed to the iteration and the calculation), which is simple, and potentially so simple that you don’t need to test it.

This refactoring means that instead of requiring 3 tests (for 0, 1, and multiple items in the list), there is now just one. This reduces the number of paths through the code from 2^9 to 2^7, or 128. This is already a lot better, but still too many to test!

• Refactored Calculation
• Refactored Test

Introduce Mockable Abstractions

The details of the inflation calculation aren’t shown in the example, but they are reasonably complicated.

To avoid this we can change the code to accept an InflationCalculator. The inflation calculation always uses the same date_of_financial_close, inflation_rate and inflation_mode, which means that it can take these in the constructor. This in turn means that the main calculation no longer requires the inflation_rate and inflation_mode.

Then in the tests we can create a mock InflationCalculator. This could for example always return a value of 2, which makes the overall calculation a lot simpler.

It is also easy to imagine that inflation calculations will happen in other parts of the code, so the abstraction will be widely useful.

This step means that instead of 4 conditional branches in the inflation calculation, there is now just one. This reduces the number of paths through the code from 2^7 to 2^3, or just 8. It is also necessary to test the InflationCalculator, and this has 4 branches, so needs 4 tests, but this still only makes 12 tests in total. Yay for loosely coupled code!

We now have a feasible number of tests to write, but including the calculation in the test is still going to be very cumbersome. Luckily we still have some refactorings left up our sleeve.

• Refactored Calculation
• Refactored Test

Test Conditional Branches in Isolation

The code branches based on certain conditionals. We can simply make some of these conditionals False, and then test each branch of the code in isolation. This way, each test only has to include the calculation for the branch that it is concerned with.

For example, we can set in_selling_mode to be False and step.start_of_step to be different to date_of_financial_close. This makes the test simple enough that it is feasible for it to perform the same calculation that the code does.

This in turn means that the test clearly communicates that the turbine_cost_including_margin should be the turbine_costs * fraction_of_spend * inflation. This helps readers understand the calculator, and achieves the goal of Tests as Documentation

At the moment the test is still quite long. However, because we are only testing a small part of the calculation, a lot of this information is now irrelevant (the any_double variable). This means we can now create a Test Data Builder, or use helper functions to make things more concise. We will see an example of this in the next refactoring.

• No Change to Calculation
• Refactored Test

Test Values in Isolation

“Test Conditional Branches in Isolation” is a useful technique, but it can still leave us with some complications if a value is calculated / updated in multiple branches.

A good example of this is balance_of_plant_cost_including_margin, which is set initially, and then updated in the if (in_selling_mode) branch.

Testing balance_of_plant_cost_including_margin in isolation allows the test to concentrate just on this one value / calculation, which means that a lot less setup is required. The Test Data Builder pattern hides Irrelevant Information, and the test becomes more concise and expressive.

Including the calculation continues to make the test clearly communicate its intent and act as documentation. Interestingly the test calculation code is no longer an exact copy of the SUT calculation code, as it already knows that it is in_selling_mode, so doesn’t need a conditional statement. This means that the test is simpler than the code, which helps avoid the Obscure Test smell.

• No Change to Calculation
• Refactored Test

Test Partial Values in Isolation

Sometimes the calculation of individual values is very complex, and can’t be split up by conditional branches. This can make it challenging to include the calculation in the test. construction_profit is a reasonable example of this, which is calculated as follows:

step.turbine_cost_including_margin = 
 turbine_costs * inflation * fraction_of_spend;

step.balance_of_plant_cost_including_margin = 
 balance_of_plant_costs_at_financial_close * inflation * fraction_of_spend;

step.construction_profit = 
 -1 * 
 (step.turbine_cost_including_margin + step.balance_of_plant_cost_including_margin) *
 epc_margin

inflation, fraction_of_spend and epc_margin have a multiplicative effect, so if we set them to 1, they won’t have any effect and we can easily write a test for the rest of the logic.

step.turbine_cost_including_margin and step.balance_of_plant_cost_including_margin have an additive effect, so if we set them to 0, again they won’t have any effect and we can easily write a test for the rest of the logic.

Testing just a portion of construction_profit in isolation allows the test to concentrate just on this part of the calculation, which, as before, makes the test shorter and simpler, and avoids the Obscure Test smell.

• No Change to Calculation
• Refactored Test

Introduce the Blackboard Pattern

The Blackboard pattern is a more involved technique, and is often badly understood. But essentially it involves using a “blackboard” to break the dependencies within complicated calculations.

In this example the construction_profit depends on turbine_cost_including_margin and balance_of_plant_cost_including_margin, which are themselves calculated from the inputs. This makes testing harder.

In order to test the construction_profit you essentially also have to test turbine_cost_including_margin and balance_of_plant_cost_including_margin.

When we introduce the blackboard pattern, one calculator writes the turbine_cost_including_margin and balance_of_plant_cost_including_margin to the blackboard, and when calculating the construction_profit we read these values from the blackboard.

This breaks the connection between the two things, so when testing we can just add values for turbine_cost_including_margin and balance_of_plant_cost_including_margin to the blackboard.

• Refactored Calculation
• Refactored Test

Conclusion

When testing calculations, it is important to include the calculation in the tests. This avoids the Hard Coded Test Data smell, allows the tests to clearly express their intent and achieve Tests as Documentation.

The refactorings described in this article allow you to do this while also making sure that the tests are concise and understandable.

In this article we will talk through the most common types of ‘Big O’ notation, using birthday cakes to illustrate the concepts. We'll pretend we're hosting a party, and need to determine how many cakes to bake based on how many people attend.

O(1) - Constant Time

For the Constant Time example, no matter how many people come to the birthday party, you only make one cake. So the cake making time stays constant.

Note that Big O notation doesn’t specify how long the Constant Time is (maybe it takes 1 hour to make the cake, maybe it takes 4 hours). It just states that the time taken doesn’t increase with the number of guests.

A real world example of an O(1) operation is accessing an array by its index. It is just as quick to retrieve an element from a 10 item array as it is to retrieve an element from a 1 million item array.

O(log n) - Logarithmic Time

For the Logarithmic Time example, the birthday cakes are used as a way to incentivise people to turn up to the party on time.

The first person to arrive gets a cake all to themselves. Then the next 2 people to arrive share a cake. Then the next 4 people all share a cake, and so on.

So a 1 person party requires 1 cake. A 2 or 3 person party requires 2 cakes. A 4 - 7 person party requires 3 cakes, and a 8 to 15 person party requires 4 cakes. In general an ‘n’ person party requires log2(n) cakes.

The most common real world example of an O(log n) operation is a binary search of an ordered array.

This algorithm looks at the middle of an array and sees if the value is lower or higher than the one it is looking for. Since the list is ordered, it then knows which half of the array the target value is in.

It then repeats the process with that half of the array. So for a 16 item array, the first iteration narrows down the search to 8 items, then 4 then 2 and then 1, for a maximum of 4, or log2(16), iterations over all.

O(n) - Linear Time

For the Linear Time example, each guest gets their own cake. If ‘n’ people come to the party, you need to make ‘n’ cakes. So the time taken is related to the number of guests.

Again Big O notation doesn’t specify how long the time is (maybe it takes 1 hour to make the cake, maybe it takes 4 hours), it just states that the time increases linearly with the number of guests.

A real world example of an O(n) operation is a naive search for an item in an array. In a 10 item array, at worst you will have to look all 10 items to find the one you want. But for a 1 million item array, you may have to look at all 1 million.

Of course, you might find the solution sooner, but Big O notation specifies the maximum amount of time that an algorithm will take.

O(n^2) - Quadratic Time

For the Quadratic Time example, each guest gets their own cake. Additionally, each cake has the names of all the guests written on it, with some delicious icing.

In this case a 1 person party has one cake with one name on it. A 2 person party has two cakes, both with two names on (4 names total) and a 3 person party has three cakes, all with three names on them, which is 9 names in total.

In general, an ‘n’ person party requires n*n names to be written (also known as n squared, or n to the power of 2), so the speed of making cakes (and writing all the names) is related to the square of the number of guests.

A real world example of an O(n^2) operation is a naive search for duplicates in an array. In this scenario, you loop through all the items in the array, and for each of those items, loop through the array again to see if there are any matches.

For a 10 item array, the outer loop has 10 iterations, and for each of those there are 10 iterations of the inner loop, for 100 in total. For a 1 million item array, it is 1000 billion.

There is a more general case of O(n^2), where instead of the time being relative to n to the power of 2 (n^2), it is relative to n to the power of c (n^c). This is usually called Polynomial Time.

O(n!) - Factorial Time

For the Factorial Time example, the guests take part in a Pétanque competition, and the winner takes home the cake.

There is a slight issue, though, in that the player that takes the first turn is at a disadvantage. To help level things out, many games are played, so that each permutation of guests is covered and everyone gets to go first. All these permutations are written on to the cake, again with some tasty icing.

This means that a 2 person party has two games, as each guest takes it in turn to go first. A 3 person party has 6 games (if we imagine that the guests are Anna, Brian and Chris, then the permutations are ABC, ACB, BAC, BCA, CAB, CBA).

In general, an ‘n’ person party requires n!, or n factorial games, so the speed of making the cake is related to the this.

n! is calculated by multiplying all whole numbers from n down to one “n (n - 1) (n - 2) … 2 1”. So for the 2 person party it is 2 1, or 2. For the three person party it is 3 2 * 1, which is 6.

Real world examples of O(n!) operations are anything that requires analysing a list of permutations, such as the traveling salesman problem.

Conclusions

Hopefully the birthday cakes have made ‘Big O’ notation easier to digest! The graph below is also a good memory aid, showing the relative speeds of the algorithms (if there is a choice, then you want the faster one!)

There are quite a few other ‘Big O’ notations, such as O(n log n) and O(c^n) but they all follow the same pattern. If you want to learn more about it, check out this article.

And what if it could be in markdown format, so it would be committed along with the rest of your code, and be shown on GitLab / GitHub?

Sounds pretty cool, right? Let's see how it's done.

Context

People like Simon Brown do a great job of convincing me that I don't have enough documentation for my projects. And that the documentation should be up to date, and show concise information at a variety of levels of abstraction.

I would love to work on a code base with documentation like that.

The Problem with Documentation

I have read a good number of books and articles about software architecture and related things. But I have never been able to summon up enough energy, or enough political capital, to be able to create documentation to this standard. Let alone keep it up to date.

So, for my situation at least, I need a way of creating and updating documentation automatically.

I would also like to store the diagrams "as code", so that they can be checked in to the repository. This way, changes to them can be easily seen and discussed in pull requests and other code reviews.

There are many tools that can generate build time dependency diagrams from code, and I have used quite a few of them.

But the problem seems to be that these diagrams always look like spaghetti, even when the code is good. And they are complex to set up.

It seems to be very hard to get the level of detail right. There is no way to show related code in logical groups for high level diagrams. There is also no way to pick out code relationships that are specific to a particular context for low level diagrams.

They also give you no information about the run time relationships of the code, which is usually a bigger issue than the design time relationships.

A solution

To capture run time relationships, generating diagrams from running code is the only option. And we already have plenty of code that is executed regularly, in the form of tests.

Repositories should already have a good suite of tests (unit, integration and end to end, for example), and each test should be relatively short and simple.

These tests should already embody logical groupings of code, and sensible levels of abstraction. So they are an excellent candidate for generating documentation.

The solution involves instrumenting the code imported by a test. This instrumented code then keeps a record of the run time call hierarchy, and is able to write the results as a Mermaid markdown diagram (tecnhically a sequence diagram).

The code below (a test from the python package) shows how it works.

For each existing test you create a "wrapper" test, which is responsible for initialising the call hierarchy and saving the diagram. If you have a lot of tests you might want to introduce a decorator to save repetition.

from docs_from_tests.instrument_call_hierarchy import instrument_and_import_package, instrument_and_import_module, initialise_call_hierarchy, finalise_call_hierarchy
from samples.hello_world_combiner import HelloWorldCombiner

# you can instrument entire packages / folders at once like this
instrument_and_import_package(os.path.join(Path(__file__).parent.absolute(), '..', 'samples'), 'samples')
# You can instrument individual modules like this
# instrument_and_import_module('tests.blah')

# this is a wrapper around the test that also outputs the documentation / sequence diagram
def test_hello_world():
    # the initialises the recording of the call hierarchy
    initialise_call_hierarchy('start')

    # This runs the actual test
    _test_hello_world()

    # this finalises the call hierarchy and returns the root
    root_call = finalise_call_hierarchy()

    # this returns a sequence diagram of the call hierarchy
    sequence_diagram = root_call.sequence_diagram(
        show_private_functions=False,
        excluded_functions=[
            'HelloWorldCombiner.__init__',
        ]
    )

    # this writes out the markdown to disk    
    sequence_diagram_filename = os.path.join(os.path.dirname(__file__), '..', 'doc', 'top-level-sequence-diagram.md')
    Path(sequence_diagram_filename).write_text(sequence_diagram)

# this is the original / source test
def _test_hello_world():
    assert HelloWorldCombiner().combine() == 'Hello world'

Running pytest on this code will result in the test being run, and the markdown "diagram as code" (below) being created in the doc directory:

sequenceDiagram
  start->>HelloWorldCombiner.combine: calls x1
  HelloWorldCombiner.combine->>hello: calls x1
  hello-->>HelloWorldCombiner.combine: returns str
  HelloWorldCombiner.combine->>world: calls x1
  world-->>HelloWorldCombiner.combine: returns str
  HelloWorldCombiner.combine-->>start: returns str

This renders as the following diagram:

Changes to the diagram will show up in Git and be committed along with the code that caused the change. This means that the change to the code and the change to the diagram are linked and can be seen together.

Private methods would usually be excluded (although it is optional), and you can exclude other functions so that the graph looks as desired.

Because the call hierarchy is stored in a tree structure, excluding a function also excludes all the functions beneath it.

Code quality

Hopefully you already have tests at appropriate levels of abstraction (classically you would have unit, integration and end to end). This makes it easy to create diagrams at these levels.

If not, then the desire to create good diagrams should guide you towards also creating good tests.

Sometimes the diagrams will look a bit crazy, and you may end up ignoring a lot of functions. This is a clue that the code could probably be made simpler. And in this case the desire to create good diagrams should guide you towards simplifying the code.

Conclusion

Hopefully this will inspire you to create and maintain the documentation that your teammates and your future self will thank you for! It's all quite easy to do.

All the functionality is in a Python package (docs-from-tests), and there is an example repo that demonstrates how to use it.

I was at a software craftsmanship meetup recently, where we formed pairs to solve a simplified Berlin Clock Kata. A Berlin Clock displays the time using rows of flashing lights, which you can see below (although in the kata we just output a text representation, and the lights in a row are all the same colour).

Initial Test Driven solution

Most pairs used inside out TDD, and there were a lot of solutions that looked something like this (complete code available on GitHub).

def berlin_clock_time(julian_time):
    hours, minutes, seconds = list(map(int, julian_time.split(":")))

    return [
        seconds_row_lights(seconds % 2)
        , five_hours_row_lights(hours)
        , single_hours_row_lights(hours % 5)
        , five_minutes_row_lights(minutes)
        , single_minutes_row_lights(minutes % 5)
    ]

def five_hours_row_lights(hours):
    lights_on = hours // 5
    lights_in_row = 4
    return lights_for_row("R", lights_on, lights_in_row)

# ...

This type of solution drops out naturally from applying inside out TDD to the problem. You write some tests for the seconds row, then some tests for the five hours row, and so on, and then you put it all together and do some refactoring. This solution does expose some of the domain concepts at a glance:

• There are 5 rows
• There is one second row, 2 hour rows and 2 minute rows

Some more concepts are available after a bit of digging, but aren't immediately obvious. The rows are made up of lights that can be on (or presumably off), and that the number of lights on is an indication of the time.

However there are some big parts of the problem that are not exposed. And since I haven't yet explained it, you probably don't know exactly how the Berlin Clock works yet.

Elevate the concepts

To improve this we can bring some of the details that are buried in the helper functions (such as get_five_hours) closer to the top of the file. This brings you to something like the following (complete code available on GitHub), although the downside is that it breaks nearly all of the tests. Solutions like this are rarer on GitHub, but do exist.

def berlin_clock_time(julian_time):
    hours, minutes, seconds = list(map(int, julian_time.split(":")))

    single_seconds = seconds_row_lights(seconds % 2)
    five_hours = row_lights(
        light_colour="R",
        lights_on=hours // 5,
        lights_in_row=4)
    single_hours = row_lights(
        light_colour="R",
        lights_on=hours % 5,
        lights_in_row=4)
    five_minutes = row_lights(
        light_colour="Y",
        lights_on=minutes // 5,
        lights_in_row=11)
    single_minutes = row_lights(
        light_colour="Y",
        lights_on=minutes % 5,
        lights_in_row=4)

    return [
        single_seconds,
        five_hours,
        single_hours,
        five_minutes,
        single_minutes
    ]

# ...

This improves the concepts that are now exposed at a glance:

• There are 5 rows
• The seconds row is a special case
• There are 2 hour rows and 2 minute rows
• The rows use different colour lights
• The rows have a different number of lights

This is pretty good, and is already better that most of the solutions out there. However, it's still a bit mysterious how the rows are related to each other (there are 2 rows to display the hours and the minutes, so presumably these are linked). It's also not obvious what amount of time each light represents.

Name implicit concepts

At the moment some of the concepts (such as the amount of time each light represents) are implicit in the code. Making these explicit, and naming them, forces us to understand them and to embed that understanding in the code.

In order to make the amount of time each light represents explicit, it seems like it would be sensible to pass a time_per_light value to row_lights. This means we have to push the calculation of lights_on down into row_lights.

This in turn makes it obvious that there are two kinds of rows: one related to the quotient (\\) of the time value, and one related to the remainder / modulus (%). If we look at the quotient case, we see that the 2nd parameter to the operation is the time_per_light, which is 5 in both cases (5 hours in one case and 5 minutes in the other).

This allows us to write these rows like this:

five_hour_row = row_lights(
    time_per_light=5,
    value=hours, 
    light_colour="R",
    lights_in_row=4)

If we now turn our attention to the remainder case, we realise that time_per_light is always singular (one hour or one minute), as it is filling in the gaps in the quotient case.

For example, the five hours row can represent 0, 5, 10, 15, or 20 hours, but nothing in between. In order to represent any hour, there must be another row to represent +1, +2, +3 and +4. This means that this row must have exactly 4 lights, and that each light must represent 1 hour.

This implies that the remainder case is dependent on the quotient one, which most people would describe as a parent / child relationship.

With this knowledge in hand, we can now create a function for the child remainder rows, and the solution now looks like this (complete code on GitHub):

def berlin_clock_time(julian_time):
    hours, minutes, seconds = list(map(int, julian_time.split(":")))

    return [
        seconds_row_lights(
            seconds % 2),
        parent_row_lights(
            time_per_light=5,
            value=hours, 
            light_colour="R",
            lights_in_row=4),
        child_remainder_row_lights(
            parent_time_per_light=5,
            value=hours,
            light_colour="R"),
        parent_row_lights(
            time_per_light=5,
            value=minutes, 
            light_colour="Y",
            lights_in_row=11),
        child_remainder_row_lights(
            parent_time_per_light=5,
            light_colour="Y",
            value=minutes)
    ]

# ...

A quick glance at this code now reveals nearly all the domain concepts

• The first row represents the seconds and is a special case
• On the second row each "R" light represents 5 hours
• The third row shows the remainder from the second
• On the fourth row each "Y" light represents 5 hours
• The fifth row shows the remainder from the fourth

This took something thinking about, which will have cost us some time / money. But we increased our understanding of the problem while we did it, and most importantly we embedded that knowledge in to the code. This means that the next person to read the code will not have to do this, which will save some time / money. Since we spend about 10 times longer reading code than we do writing it, this is probably a worthwhile endeavour.

Embedding this understanding has also made it harder for future programmers to make mistakes. For example, the concept of parent / child rows didn't exist in earlier examples, and it would be easy to mismatch them. Now the concept is plain to see, and the values are mostly worked out for you. It is also easier to refactor to support new clock variants, for example where lights in the first hours row represent 6 hours.

How far should you take it?

There are things we can do to take this further. For example the parent_time_per_light of a child row must match the time_per_light of its parent, and there is nothing enforcing this. There is also a relationship between time_per_light and lights_in_row for the parent rows, and again it is not enforced.

However, at the moment we are only required to support one clock variant, so these probably aren't worth doing. When a change is required for the code, we should refactor so that the change is easy (which might be hard) and then make the easy change.

Conclusions

Embedding domain concepts in code requires thought and skill, and TDD won't necessarily do it for you. It takes longer than a naive solution, but makes the code easier to understand, and will very likely save time in the medium term. Time is money, and finding the right balance of spending time now versus saving time later is also an important skill for a professional programmer to have.

This is usually a fertile breeding ground for bugs born out of miscommunication / misunderstanding, and fixing these bugs often introduces new ones, for the same reasons. In the end it becomes code that no one really understands or wants to touch.

Example of undescribable code

It is easy to think that code is already a written language. If it looks simple, it should be easy to read, speak and listen to. However, this is not always the case.

Below is a common solution to deciding whether a year is a leap year.

(divisibleBy(4) and not divisibleBy(100)) or divisibleBy(400)

This is not overly complicated code. It calls a functions 3 times, has 3 operators (and, or, not), and has two levels of nesting.

However, if you take a second to try and describe the algorithm in words I think you will find it to be a struggle.

Maybe “A year is leap year if it is divisible by 4 and not divisible by 100, or divisible by 400”?

The trouble with this is that the code has brackets, but the words do not. So they cannot adequately describe the condition, and whether “or divisible by 400” applies to “divisible by 4” or “not divisible by 400”. You could try some hand waving and gesturing to get around this, or vary the length of pause between the statements, but hopefully it’s obvious that there is a lot of potential for error.

Refactoring to describable code

Instead we can start by describing the condition with words, and then make the words as clear and concise as possible. We might start with this:

“400 years is a special case. If a year is divisible by 400, then it is a leap year. 100 years is also a special case. If a year is divisible by 100 then it isn’t a leap year, unless it is also divisble by 400, the 400 year special case takes priority. If there are no special cases, then the year is a leap year if it is divisible by 4.”

This is clear, but isn’t concise, so we would probably want to shrink it a bit:

“If a year is divisible by 400, then it is a leap year. Otherwise if it is divisible by 100 then it is a normal year, otherwise it is a leap year if it is divisible by 4.”

If we turn these words in to code, we probably get something like the following:

    if divisbleBy(400):
        return LeapYear
    elif divisbleBy(100)
        return NormalYear
    elif divisbleBy(4):
        return LeapYear
    else:
        return NormalYear

Conclusions

Hard to understand code is a daily occurrence for virtually all programmers. We can help ourselves and our co-workers by writing code that is easy to describe in words.

And the great thing is that doing so is actually easier than writing code any other way, as there is no mental mapping / wasted mental effort. The only “trick” is to describe the algorithm in words, and then write code to match the words.

In many organisations, the algorithm will already be described in words, as part of acceptance tests or user stories, which will improve productivity even further.

Netlify functions provide a very convenient way of working with AWS Lambdas, and they have an impressive array of example use cases, such as sending emails, processing payments and logging.

This example reads secrets from environment variables (to avoid them being exposed in the browser), but it is mostly generic, and can be adapted for other use cases easily.

Step 1 - Prerequisites

• Create a repository for the code, probably on GitHub
• npm i -g elm
• npm install -g netlify-cli
• npm i -g create-elm-app

Step 2 - Create vanilla Elm app

• create-elm-app elm-app-with-netlify-function
• cd elm-app-with-netlify-function
• elm-app start

This should start a development server and load the app in your browser.

You can look at this commit in the companion repository to check that everything is ok.

Step 2 - Deploy on Netlify

• npm init (and fill in sensible values)
• npm i create-elm-app --save-dev (this adds create-elm-app to package.json, which is used by netlify)
• Push the code to GitHub

You can see the results of this at this commit in the companion repository

• Log in / Sign up / register with Netlify
• Create a new site on Netlify
• Choose your repository
• Set the "Build command" to elm-app build
• Set the "Public directory" to build
• Click on "Deploy Site"

Netlify will now deploy the site, installing the dependencies specified in package.json, then running elm-app build and then serving the dist directory.

From now on, Netlify will attempt to deploy the latest code every time you push to GitHub.

Step 3 - Link Netlify Dev

• netlify login
• netlify link and choose the “Use current git remote url” option
• Add “./netlify” to .gitignore
• Add a netlify.toml file (from this commit in the companion repository)
• netlify dev

This should start a local development server and load the app in your browser, in a similar way to step 1.

Step 4 - Add a netlify function

Run netlify functions:create to create a new netlify function. Choose the “js-token-hider” template, and name it "call-api".

This will create a javascript file for the function, and a package.json for its dependencies in “functions/call-api”.

Replace functions/call-api/call-api.js with this one in the companion repository

Now if you run netlify dev, the function will be served as well as the app, albeit on different ports. You can view the function in the browser to check that it is working (probably at http://localhost:34567/call-api or http://localhost:34567/.netlify/functions/call-api)

Step 5 - Call the netlify function from Elm

Install depdencies

• elm install elm/json
• elm install elm/http
• elm install krisajenkins/remotedata

Update Main.elm to call the function and display the results (from the companion repository).

Instruct create-elm-app to proxy api calls to the function, by adding elmapp.config.js, as shown in the companion repository.

At this point, thee application runs, and successfully calls the api, but there are no secrets / environment variables yet, so the UI shows an error.

Step 6 - Add the secrets

Go to the “Site Settings” - “Build & Deploy” - “Continuous Deployment” - “Environment Variables” section on the Netlify website for your application.

Add environment variables for API_TOKEN and API_URL

Now when you run ‘netlify dev’ the app should now load in the browser and call the locally hosted netlify function, which will return the API_TOKEN and API_URL environment variables that you set on Netlify.

The same should be true on the live deployment on Netlify. You may need to “Trigger Deploy” manually on Netlify, so that it uses the new environment variables.

You can see the deployment of the companion repository at https://netlify-functions-with-elm.netlify.com

Conclusion

Netlify / serverless functions are extremely useful for creating / connecting to the backend services that your front end needs. They are also very east to set up, as this artcile (hopefully!) shows.

Create-elm-app is a great tool for developing Elm applications, and its simple proxy feature works well when developing Netlify functions.

Netlify Dev is great for replicating the production Netlify setup when developing locally (in this case by automatically providing the environment variables).

In Elm, there are a lot of great ways to scale the Model, and update, but there is more controversy around scaling the view. A lot of the debate is around Reusable Views versus Components. Components are not recommended, but a lot of people are still advocating for them. This article presents an idea that hopefully strengthens the argument for Resuable Views.

In almost all cases, the scaling problem comes down to enforcing consistency, which usually means allowing child views to make some adjustments to the master view, while at the same time not allowing child views to make a mess.

I will be using Richard Feldman's excellent Real World app (specifically written to demonstrate scaling in Elm) as an example, as it is contains a lot of current best practice techniques, it is well known (2000+ stars and 300+ forks) and Richard is a well known Elm expert.

I will be suggesting some improvements to this code, so I want make a clear at this point that I mean no disrespect by this (I would bet large sums of money that he did it in about one tenth of the time it would have taken me!). You could also argue that the problems are small and not worth fixing. Ultimately, this decision is yours, but by the end of the article I hope to persuade you that there are problems, and that they are fixable if you think it is worthwhile.

Master view functions with conditionals

One option is to define a master view function. This function takes care of shared concerns, like the header bar and overall layout. Then it calls child view functions depending on the current view and / or has parameters to control child specific behaviour.

This works, but can quickly lead to:

• An explosion of parameters, potentially forcing your child views to return a lot of things they don't care about.
• A mixing of responsibilities between master and child views.
• Extra code and duplication.

In the Real World App, a parameter of type Page is passed to the master view so that it can render a navbar link as active. There is a large case statement that uses this parameter to work out what which link is active, and it would be a lot easier for the child just to specify this.

The line below shows the master view passing Page.Home, which has to match up with Home.view home. This is easy to get wrong, there is no help from the compiler or type system, and really it is the responsibility of the child view the specify this.

viewPage Page.Home GotHomeMsg (Home.view home)

There is some duplication when creating the NavBarLink Html, and the linkTo function will accept any Html, although only very particular Html is valid.

Convention and trust

Another possibility is for child views to be responsible for keeping shared elements consistent, by convention and trust.

Arguably this also happens in the Real World App. The Home, Article and Profile views all have the concept of a banner. The banner is different in each view, but presumably is meant to be a consistent and recognisable visual element (essentially, it's the title / header for the view). The views don't share any code for these banners, and as a result of this they are not the same size or colour. You could theoretically try and enforce a convention using tests, but it would be difficult, and probably not worthwhile.

Helper functions

Another possibility is for child views to be responsible for keeping shared elements consistent, but by using some helper functions. This is definitely a step forward, and is probably the most common solution I see in the wild. The functions can go in the same file and be next to each other. This makes it easier to see that they are related and are representing the same visual element, and easier to make them consistent.

However, there are still some drawbacks. The main one is that the child views have to know to use the helper functions, and there is nothing enforcing this. This isn't a huge deal when you only have one shared element and one function to call, but as applications get bigger, you end up with a combinatorial explosion of differences in the shared visual elements. Most people tame this by providing a number of small, focused functions for the various differences. Then the child view has to know about all these functions, and how to compose them, and there no help from the compiler.

Again, this arguably occurs in the Real World App: for example in this part of the Profile.view function, which needs to know how to use the viewTabs, Feed.viewArticles and Feed.viewPagination helper functions, and what Html they need to be contained in.

Scaling with Master View Types

In order to overcome these problems, I propose using a Type to define your site structure (I rather pompously call this a "Master View Type"). Child views then return this type, and the master view takes it as a parameter and returns the html.

For the Real World App examples we have been looking at, the Master View Type is below (Viewer is the person viewing the page in the Real World App). You could arguably have more general banner types here, such as AvatarBanner, or even IconBanner (instead of ViewerBanner) depending on your domain.

type alias Page =
    {   activeNavBarLink: NavBarLink
        , banner: Banner
        , body: Html Msg
    }

type Banner =
    TextBanner TextBannerProperties
    | ViewerBanner Viewer
    | ArticleBanner Viewer ArticlePreview

type NavBarLink =
    NavBarLink NavBarLinkProperties

To demonstrate this, I have create a repository with just the Header and Banner parts of the Real World App and then created a new repository after refactoring to use a Master Page Type, NavBarLink Type and Banner Type. You can peruse the code to get a feel for how it works.

To my mind, using a Master Page Type has the following benefits:

• Writing the master view code is easier
• Writing the child view code is easier
• Communication and understanding are improved, as UI concepts now have names
• Theming / redesigning a site is a lot easier
• Elm packages can provide UI templates

The master view can precisely define what it will accept / support via the types, with union types and opaque types. Non supported combinations can be made unrepresentable or uncreatable.

In my example repository the NavBarLink type is opaque, so it is only possible to create supported NavBarLinks (home, article and viewer). In a similar way Banner is a union type, which means that only supported variants can be represented.

It would be possible for a programmer to simply change these files, but a proficient programmer would recognise the patterns and follow them. If this isn't enough and you are feeling paranoid, then you can require stricter code review on such files, potentially taking advantage of CODEOWNERS functionality on GitHub and GitLab. In the extreme you can provide the modules via an elm package, and restrict push access to the underlying repository.

Child views don't have to do anything more than create an instance of the types. The helper functions all return types, so it's easy to see which functions can be used in a particular context, and is impossible to use functions in the wrong context. For example, if a function returns a HeaderBarLink, it is impossible to mistakenly use this function to create a link in the FooterBar, or elsewhere on the page. Child views can also leave some of the complexity to the master view. For example, the child view can define a list of options to choose from, and the master view can render this using buttons, a drop down list or an autocomplete list, depending on the number of options.

The master page type also provides names for UI concepts, which can then be discussed. For example, a designer could say "Let's move the NavBarLinks to the left hand side", and everybody would know what they meant. A product owner could say "Let's create a new page with an IconBanner, and we'll use the current weather api for the icon" and again, everybody would know what they mean. You can look at this excellent thoughtworks article for more details of this.

Since the responsibility for turning the Master View Type in to html is all in the same place, it is easy to make drastic changes to the look and feel of a website, and to do theming. These changes and themes can alter the Css and the Html, which is something that the normal theming techniques just can't do. Pragmatically, your Master View Type will often have a body: Html Msg property (to allow child views complete flexibility on the child specific parts of the page) so there would still be some sprawling code to fix up, but it will definitely be a lot easier.

Finally, it opens up possibility of providing ready made themes and site layouts as packages. This would allow you to just do the following to get a working app, complete with layout and styling:

• create-elm-app
• elm install elm-bootstrap-starter-template
• Write some code to create the Master Page Type
• elm-app start

Companies could create packages like these to ensure a consistent look and feel across their applications. Open source designs and layouts could emerge and become commonplace, similar to the way that Bootstrap has revolutionised html and css design. Developers with limited design skills (like me) could concentrate on the the bits they are best at (the logic), but still produce produce elegant websites using these packages.

To demonstrate this I have created a bootstrap starter master view package. It mimics the layout and design of the bootstrap starter template. I have then used this package in a demo elm application. You can browse the demo application to see how it looks, and view the source to see how it works.

All these advantages come at a small to negative cost. There is a little more code for the new types, but some duplication is removed. You can view the source of the Real World App repositories from before and after refactoring to use a Master Page Type for the full details.

Conclusions

Master View Types bring a lot of benefits (view code is easier to write and maintain, UI concepts are named and UI packages are possible) for little or no cost. They should improve the code of any Elm application that has issues around enforcing consistency (while allowing flexibility) in their view code, which in my experience is most medium and large applications.

I didn’t want to create an on-premises server. For us, it would make it impossible to use cloud-based build servers, and it is another moving part that can go wrong. There are also potential issues with fine-grained security and speed. (We have a worldwide team, so serving the content via a CDN would be helpful.)

I didn’t want to force the team to create accounts with another provider. They already have Active Directory and GitHub accounts. It is an annoyance for them and creates a governance burden for me.

Sadly, I couldn’t find such a service. GemFury is excellent but doesn't support GitHub authorization (at the team / organisation level) and Packagr doesn’t support GitHub authorisation at all. MyGet is also excellent, it does allow me to use GitHub authorization, but doesn’t host Python packages. Azure DevOps has something that looks promising, but it’s in private beta at the moment.

Happily, this is possible using cloud Git repositories such as GitHub, GitLab and BitBucket.

Pip can install packages from Git

I have hosted a Python package on GitHub (python_world), which you can install with the following command (make sure you trust me before running this command and installing my code on your computer).

pip install git+https://github.com/ceddlyburge/python_world#egg=python_world

Pip provides options to install from head, from a branch, from a tag or from a commit. I usually tag each release and install from these tags. See the pip install documentation for full details.

This repository is public, but it works just the same with a private repo, as long as you have permission. There is no special magic (it's a vanilla Python package) and Setup.py does most of the work as normal.

If you are new to creating Python Packages, the Packaging Python Projects tutorial is worth a quick read.

Setuptools can also install dependencies from Git

Setuptools is how most people create Python packages.

I have hosted another package on GitHub python_hello, which depends on python_world. (I’m sure you can see where this is going.)

The relevant bits from setup.py are below. install_requires specifies that python_world is a required dependency and tells Setuptools where to find it.

install_requires=[
    'python_world@git+https://github.com/ceddlyburge/python_world#egg=python_world-0.0.1',
]

You can install this package using the command below. It will also download the dependent python_world package.

pip install git+https://github.com/ceddlyburge/python_hello#egg=python_hello

This links to a specific version of python_world, which is a shame as it means that pip can't do any dependency management (such as working out an acceptable version if multiple things are reliant on it). However, by the end of this article, we will have removed the need for the specific link.

Python environments

As everyone who has used Python without an environment knows, environments save a lot of frustration and wasted time. So we need to support those.

I have created a repo (use-hello-world) that defines python_hello as a dependency in requirements.txt for Virtualenv, and environment.yml for Conda.

If you download the repo, you can install the dependencies into a virtualenv with the following command.

pip install -r requirements.txt

If you are using conda you can use this command:

conda env create -n use-hello-world

PyPi Index

So far we are able to install packages from our private Git repositories. These packages can, in turn, define dependencies to other private repositories. There still isn’t a PyPi server in sight.

We could stop at this point. However, the syntax for defining dependencies is a bit mysterious. It would be difficult for the team to discover which packages are available, and we are linking to specific versions of dependent packages, instead of letting pip manage it.

To fix this we can set up a PyPi index that conforms to Pep 503. This specification is quite simple, and I have just created the index by hand. If this becomes too cumbersome I can generate it from the GitHub API.

I created this PyPi Index using GitHub Pages. There are equivalent things for GitLab and BitBucket. You can see that the source code is very simple. GitHub Pages sites are always public (and there is probably no sensitive information in your index). However, if you need them to be private you can use a service such as PrivateHub.

One thing to look out for is the name normalisation of the specification. This requires the python_hello package information to be present at python-hello/index.html (note the change from an underscore to a dash).

Now that we have a PyPi server, we can install packages using the command below.

pip install python_hello --extra-index-url [https://ceddlyburge.github.io/python-package-server/](https://ceddlyburge.github.io/python-package-server/)

So that you can see this working with environments, I have created another repo (use_hello_world_from_server) that defines the python_hello dependency using this PyPi index instead of direct GitHub Links. If you are trying it with Conda, version >4.4 is required.

At this point, we can go back and remove the direct Git link in install_requires in setup.py of python_hello (as Setuptools will be able to find it from our server).

Conclusions

Using a cloud-hosted Git provider as a PyPi server is a viable option. If you are already using one, that means that you can reuse the credentials and permissions that you already have. It will work with Cloud build servers and is likely to be provided via a CDN, so will be fast worldwide. It requires more knowledge to set up than a hosted server, but probably the same or less than hosting your own server on premises.

Hints and tips

Serving the index locally can help to troubleshoot problems (such as name normalization). It’s easy to see what requests are being made. You can use the inbuilt python HTTP server for this (python -m Http.Server -8000). This led me to find out that pip search uses postrequests, so won’t work with GitHub pages.

You can run python setup.py -install to check your pip packages locally, before pushing them to Git.