Listen Notes is a podcast search engine and database. The technology behind Listen Notes is actually very very boring. No AI, no deep learning, no blockchain. “Any man who must say I am using AI is not using True AI” :)

After reading this post, you should be able to replicate what I build for Listen Notes or easily do something similar. You don’t need to hire a lot of engineers. Remember, when Instagram raised $57.5M and got acquired by Facebook for $1B, they had only 13 employees — not all of them were engineers. The Instagram story happened in early 2012. It’s 2019 now, it’s more possible than ever to build something meaningful with a tiny engineering team — even one person.

If you haven’t used Listen Notes yet , try it now:

https://www.listennotes.com/

Overview

Let’s start with requirements or features of this Listen Notes project.

Listen Notes provides two things to end users:

• A website ListenNotes.com for podcast listeners. It provides a search engine, a podcast database, Listen Later playlists, Listen Clips that allows you to cut a segment of any podcast episode, and Listen Alerts that notifies you when a specified keyword is mentioned in new podcasts on the Internet.
• Podcast Search & Directory APIs for developers. We need to track the API usage, get money from paid users, do customer support, and more.

I run everything on AWS. There are 20 production servers (as of May 5, 2019):

The servers that run Listen Notes

You can easily guess what does each server do from the hostname.

• production-web serves web traffics for ListenNotes.com.
• production-api serves api traffics. We run two versions of API (as of May 4, 2019), thus v1api (the legacy version) and v2api (the new version).
• production-db runs PostgreSQL (primary and replica)
• production-es runs an Elasticsearch cluster.
• production-worker runs offline processing tasks to keep the podcast database always up-to-date and to provide some magical things (e.g., search result ranking, episode/podcast recommendations…).
• production-lb is the load balancer. I also run Redis & RabbitMQ on this server, for convenience. I know this is not ideal. But I’m not a perfect person :)
• production-pangu is the production-like server that I sometimes run one-off scripts and test changes. What’s the meaning of “pangu”?

Most of these servers can be horizontally scaled. That’s why I name them production-something1, production-something2… It could be very easy to add production-something3 and production-something4 to the fleet.

Backend

The entire backend is written in Django / Python3. The operating system of choice is Ubuntu.

I use uWSGI to serve web traffics. I put NGINX in front of uWSGI processes, which also serves as load balancer.

The main data store is PostgreSQL, which I’ve got a lot of development & operational experience over many years — battle tested technology is good, so I can sleep well at night. Redis is used for various purposes (e.g., caching, stats,…). It’s not hard to guess that Elasticsearch is used somewhere. Yes, I use Elasticsearch to index podcasts & episodes and to serve search queries, just like most boring companies.

Celery is used for offline processing. And Celery Beat is for scheduling tasks, which is like Cron jobs but a bit nicer. If in the future Listen Notes gains traction and Celery & Beat cause some scaling issues, I probably will switch to the two projects I did for my previous employer: ndkale and ndscheduler.

Supervisord is used for process management on every server.

Wait, how about Docker / Kubernetes / serverless? Nope. As you gain experience, you know when not to over-engineer. I actually did some early Docker work for my previous employer back in 2014, which was good for a mid-sized billion-dollar startup but may be overkill for a one-person tiny startup.

Frontend

The web frontend is primarily built with React + Redux + Webpack + ES. This is pretty standard nowadays. When deploying to production, JS bundles would be uploaded to Amazon S3 and served via CloudFront.

On ListenNotes.com, most web pages are half server-side rendered (Django template) and half client-side rendered (React). The server-side rendered part provides a boilerplate of a web page, and the client-side rendered part is basically an interactive web app. But a few web pages are rendered entirely via server side, because of my laziness to make things perfect & some potential SEO goodies.

Audio player

I use a heavily modified version of react-media-player to build the audio player on ListenNotes.com, which is used in several places, including Listen Notes Website, Twitter embedded player, and embedded player on 3rd party websites:

Embedded player on 3rd party websites

Podcast API

We provide a simple and reliable podcast API to developers. Building the API is similar to building the website. I use the same Django/Python stack for the backend, and ReactJs for the frontend (e.g., API dashboard, documentation…).

Listen API dashboard

Listen API documentation

For the API, we need to track how many requests a user use in current billing cycle, and charge $$$ at the end of a billing cycle. It’s not hard to imagine that Redis is heavily used here :)

DevOps

Machine provisioning & code deployment

I use Ansible for machine provisioning. Basically, I wrote a bunch of yaml files to specify what type of servers need to have what configuration files & what software. I can spin up a server with all correct configuration files & all software installed with one button push. This is the directory structure of those Ansible yaml files:

I could’ve done a better job in naming things. But again, it’s good enough for now.

I also use Ansible to deploy code to production. Basically, I have a wrapper script deploy.sh that is run on macOS:

./deploy.sh production HEAD web

The deploy.sh script takes three arguments:

• Environment: production or staging.
• Version of the listennotes repo: HEAD means “just deploy the latest version”. If a SHA of a git commit is specified, then it’ll deploy a specific version of code — this is particularly useful when I need to rollback from a bad deployment.
• What kind of servers: web, worker, api, or all. I don’t have to deploy to all servers all at once. Sometimes I make changes on Javascript code, then I just need to deploy to web, without touching api or worker.

The deployment process is mostly orchestrated by Ansible yaml files, and of course, it’s dead simple:

• On my Macbook Pro, if it’s to deploy to web servers, then build Javascript bundles and upload to S3.
• On the target servers, git clone the listennotes repo to a timestamp-named folder, check out the specific version, and pip install new Python dependencies if any.
• On the target servers, switch symlink to the above timestamp-named folder and restart servers via supervisorctl.

As you can see, I don’t use those fancy CI tools. Just dead simple things that actually work.

Monitoring & alerting

I use Datadog for monitoring & alerting. I’ve got some high level metrics in a simple dashboard. Whatever I do here is to boost my confidence when I am messing around the production servers.

Datadog dashboard for Listen Notes, as of Dec 2017.

I connect Datadog to PagerDuty. If something goes wrong, PagerDuty will send me alerts via phone call & SMS.

I also use Rollbar to keep an eye on the health of Django code, which will catch unexpected exceptions and notify me via email & Slack as well.

I use Slack a lot. Yes, this is a one-person company, so I don’t use Slack for communicating with human beings. I use Slack to monitor interesting application-level events. In addition to integrating Datadog and Rollbar with Slack, I also use Slack incoming webhooks in Listen Notes backend code to notify me whenever a user signs up or performs some interesting actions (e.g., adding or deleting things). This is a very common practice in tech companies. When you read some books about Amazon or PayPal’s early history, you’ll know that both companies had similar notification mechanism: whenever a user signed up, there would be a “ding” sound to notify everyone in the office.

Since launched in early 2017, Listen Notes hasn’t got any big outage (> 5 minutes) except for this one. I’m always very careful & practical in these operational stuffs. The web servers are significantly over-provisioned, just in case there’s some huge spike due to press events or whatever.

Development

I work in a WeWork coworking space in San Francisco. Some people may wonder why not just work from home or from some random coffee shops. Well, I value productivity a lot and I’m willing to invest money in productivity. I don’t believe piling time helps software development (or any soft of knowledge/creativity work). It’s rare that I work over 8 hours in a day (Sorry, 996 people). I want to make every minute count. Thus, a nice & relatively expensive private office is what I need :) Instead of optimizing for spending more time & saving money, I optimize for spending less time & making money :)

My office at WeWork

I’m using a MacBook Pro. I run the (almost) identical infrastructure inside Vagrant + VirtualBox. I use the same set of Ansible yaml files as described above to provision the development environment inside Vagrant.

I subscribe to the monolithic repo philosophy. So there’s one and only one listennotes repo, containing DevOps scripts, frontend & backend code. This listennotes repo is hosted as a GitHub private repo. I do all development work on the main branch. I rarely use feature branches.

I write code and run the dev servers (Django runserver & webpack dev server) by using PyCharm. Yea, I know, it’s boring. After all, it’s not Visual Studio Code or Atom or whatever cool IDEs. But PyCharm works just fine for me. I’m old school.

My PyCharm

Miscellaneous

There are a bunch of useful tools & services that I use to build Listen Notes as a product and a company:

• iTerm2 and tmux for the terminal stuffs.
• Notion for TODO lists, wiki, taking notes, design documents…
• G Suite for @listennotes.com email account, calendar, and other Google services.
• MailChimp for sending the monthly email newsletter.
• Amazon SES for sending transactional & some marketing emails.
• Gusto to pay myself and contractors who are not from Upwork.
• Upwork to find contractors.
• Google Ads Manager to mange direct sales ads and track performance.
• Carbon Ads and BuySellAds for fallback ads.
• Cloudflare for DNS management, CDN, and firewall.
• Zapier and Trello to streamline the podcaster interview workflow.
• Medium for the company blog (obviously).
• Godaddy and Namecheap for domain names.
• Stripe for getting money from users (primarily for API).
• Google speech-to-text API to transcribe episodes.
• Kaiser Permanente for health insurance.
• Stripe Atlas to incorporate Listen Notes, Inc.
• Clerky to generate legal documents for fund raising (SAFE) and hiring contractors who are not from Upwork.
• Quickbooks for bookkeeping.
• 1password to manage login credentials for tons of services.
• Brex for charge card — you can get incremental $5000 AWS credits, which can be applied on top of the AWS credits from WeWork or Stripe Atlas.
• Bonvoy Business Amex Card — You can earn Marriott Bonvoy points for luxury hotels and flights. It’s the best credit card points for traveling :)
• Capital One Spark for checking account.

Keep calm and carry on…

As you can see, we are living in a wonderful age to start a company. There are so many off-the-shelf tools and services that save us time & money and increase our productivity. It’s more possible than ever to build something useful to the world with a tiny team (or just one person), using simple & boring technology.

As time goes, companies become smaller and smaller. You don’t need to hire tons of full-time employees. You can hire services (SaaS) and on-demand contractors to get things done.

Most of time, the biggest obstacle of building & shipping things is over thinking. What if this, what if that. Boy, you are not important at all. Everyone is busy in their own life. No one cares about you and the things you build, until you prove that you are worth other people’s attention. Even you screw up the initial product launch, few people will notice. Think big, start small, act fast. It’s absolutely okay to use the boring technology and start something simple (even ugly), as long as you actually solve problems.

There are so many cargo-cult-type people now. Ignore the noises. Keep calm and carry on.

If you haven’t used Listen Notes yet , try it now:

https://www.listennotes.com/

We use object pool in jasync-sql to manage connections to the database. In this post, I will share how it is done in a performant, lock-free manner using Kotlin coroutines with an Actor.

An object pool has a very simple API to work with. It is a pool of objects with two methods: take() and return().

On first sight it looks like a very simple problem. The main catch here is that it has to be both performant and thread-safe, and that’s what makes it interesting and tricky to implement.

But hey! Why do we need an object pool anyway?

jasync-sql is a library to access relational databases like MySQL and PostgreSQL. Database connections are a great example of the need for object pools. The access to the database is done by obtaining a connection from a Connection-Pool, using it and returning it back to the pool.

With a connection pool we get a couple of advantages over creating connections per each SQL query:

• Reusing connections — since the overhead of initiating a connection to the database is high (handshake, etc), connection pools allow keeping connections alive, thus reducing that overhead.
• Limiting resources — creating a DB connection per user request can be overwhelming to the DB. Using a pool effectively adds a barrier, limiting the number of maximum number of concurrent connections.

Well, I am sold, but…

Isn’t a Connection Pool a solved problem in the Java world?

Yes it is a solved problem if you’re using JDBC. In that case HikariCP is an excellent choice from my experience, but there are a lot of others. In the case of jasync-sql it is not possible to use HikariCP, because HikariCP works with the JDBC API, and the jasync-sql driver is not implementing that full-fledged API, only a subset of it.

What about other Object pools in Java world?

There are numerous implementations, but it turns out that you usually find some specific requirement that was not implemented by that pool you’re using.

In our case, that requirement was non-blocking. In our pool, all operations have to be non-blocking since the library is async. For example, the take() operation in most implementations returns an object immediately or blocks until an object is ready. Our take() returns a Future<Connection>, which will be completed and continued when the connection is ready to use.

I haven’t seen such an implementation in the wild.

I really like this answer from Stack Exchange:

Is object pooling a deprecated technique?
_Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working…_softwareengineering.stackexchange.com

Another requirement that makes it hard to find an alternative is the need to try and stay compatible as much as possible with the current implementation we have.

In case you want to see other implementations you can check here:

object pool in java - Google Search
_object pool is a collection of a particular object that an application will create and keep on hand for those…_www.google.co.il

So how did we implement Object Pool?

Before we dive into the details, let’s observe other requirements from the object pool that were omitted above for clarity but are necessary details.

Interfaces

The Object pool interface looks like this:

interface AsyncObjectPool<T&gt; {  fun take(): CompletableFuture&lt;T>  fun giveBack(item: T): CompletableFuture<AsyncObjectPool<T>>  fun close(): CompletableFuture<AsyncObjectPool<T>>

}

In addition, when a pool wants to create new objects (connections) it will call the ObjectFactory. The factory has a couple more methods to handle the object lifecycle:

• validate — a method to check that the object is still valid. The method should be fast and check only in-memory constructs. For connections we usually check that the last query did not throw an exception and did not get a termination message from netty.
• test — similar to validate, but a more exhaustive check. We allow test method to be slow and access the network etc. This method is used to check that idle objects are still valid. For connections, that will be something similar to select 0.
• destroy — called to clean up the object when the pool is not using it anymore.

The complete interface is:

interface ObjectFactory<T> {  fun create(): CompletableFuture<;out T>  fun destroy(item: T)  fun validate(item: T): Try<T>  fun test(item: T): CompletableFuture<T>

}

For pool configuration we have the following properties:

• maxObjects — maximum number of connections we allow.
• maxIdle — time that we leave the connection open without use. After that time it will be reclaimed.
• maxQueueSize — when a request for a connection arrives and no connection is available, we put the request on hold in a queue. In case the queue is full (its size passed maxQueueSize) it will not wait but instead return an error.
• createTimeout — maximum time to wait for a new connection to be created.
• testTimeout — maximum time to wait for a test query on an idle connection. If it passes we will consider the connection as erroneous.
• validationInterval — on this interval, we will test if the idle connections are active and free up connections that passed maxIdle. We will also remove connections that passed testTimeout.

Original implementation

The first implementation of object pool was single threaded. All operations were sent to a worker thread that was responsible to execute them. This method is known as thread-confinement. Object creation and test operations were blocking and query execution itself was non-blocking.

This method is problematic because operations are done one after another. On top of that, there are a couple of operations that are blocking as mentioned above. There were various cases of high latency when working in some scenarios and use cases (like here for example).

As a workaround PartitionedPool was introduced. This is a workaround to the block issue with the above single-threaded approach. The partitioned pool creates multiple SingleThreadedObjectPools, each with its own worker. When a connection is requested, a pool is selected by a modulus on the thread id. The partitioned pool is actually a pool of pools ;-)

I mentioned this is a workaround since it has its own problems: you might still be blocking, but at a lower rate — plus it consume more threads and resources.

Actor based implementation

An Actor is an entity that has a mailbox. It receives messages to its mailbox and processes them one after the other. The mailbox is a sort of a channel to pass events from the outside world to the actor.

A coroutines actor employs lock-free algorithms to allow fast and performant execution of events without the need for locks and synchronized blocks.

_“wall rack filled with paper document lot” by [Unsplash](https://unsplash.com/@californong?utm_source=medium&utm_medium=referral" rel="noopener" target="_blank" title="">Nong Vang on <a href="https://unsplash.com?utm_source=medium&utm_medium=referral" rel="noopener" target="blank" title=")

You can see an elaborated explanation here.

In our case those events will be take and giveBack. In addition to those, we will have internal messages that the actor sends to itself like objectCreated etc. That allows the actor to have states that does not suffer from concurrency problems, as it is always confined to the same sequential execution. In addition the channel that passes those events is a queue that is using lock-free algorithms so it is very efficient, avoids contention, and generally has very high performance.

There is an excellent video explaining how this was implemented (note that this is “heavy” algorithmic staff):

Let’s recap what we have until now:

• An actor receives messages and processes them one by one.
• Usually messages will contain a CompletableFuture that should be completed when the actor processes it.

Messages will be completed immediately or delayed (like in case we are waiting for a connection to be created). If it is delayed the actor will put the Future in a queue, and will use a callback mechanism to notify itself when the original future can be completed.

• Message processing in the actor should not be blocked or delay the actor. If this happens, it will delay all messages waiting to be processed in the queue and will slow down the entire actor operation.

That’s why, in case we have long running operations inside the actor, we use the callback mechanism.

Let’s see more details on the use cases

Take — someone wants an object from the pool. It will send a message with a callback to the actor. The actor will do one of the following things:

• If the object is available — the actor will simply return it.
• If the pool hasn’t passed the limit of created objects — the actor will create a new object and return it when the object is ready.

In such a case, object creation can take time, so the actor will connect the callback from the object creation to the original take request callback.

• Will put the request in a queue for an available object (unless the queue is full and in that case will just return an error).

GiveBack — someone wants to give an object back to the pool (release it). This is also done by a message to the actor. The actor will do one of the following:

• If someone is waiting on the wait queue — it will borrow the object to it.
• In other cases it will just keep the object on the pool for requests to come, so the object remains idle.

Test — periodically, someone from outside will notify the actor to test connections:

• The actor will release the idle connection that hasn’t been used for a long time (it’s configurable).
• The actor will test other idle objects using the ObjectFactory. It will send a callback to the factory and mark those objects as In Use, to prevent from borrowing them until the test is completed.
• The actor will check for timeouts in tests and destroy time-outed objects.

Those are the main use cases.

Leaks

_“selective focus photography of brown faucet” by [Unsplash](https://unsplash.com/@leipuri?utm_source=medium&utm_medium=referral" rel="noopener" target="_blank" title="">Jouni Rajala on <a href="https://unsplash.com?utm_source=medium&utm_medium=referral" rel="noopener" target="blank" title=")

There can be all sort of leaks in an object pool. Some are internal bugs which I hope are easier to spot and fix, and others are objects that were taken but not returned due to some user error. In such cases, objects might remain in the “In Use” queue forever.

To avoid such cases, the “In Use” Map is using Java’s WeakHashMap. So if a user lost a connection it will be automatically removed from the map when it is cleaned by Java’s Garbage-Collector.

In addition we added a log message in such cases that says: “LEAK-DETECTED”.

That’s it!

The full Kotlin source code of the object pool is available here:

jasync-sql/jasync-sql
_Java async database driver for MySQL and PostgreSQL written in Kotlin - jasync-sql/jasync-sql_github.com

In an upcoming post I will compare performance metrics of the different implementations.

If you want to read more about Kotlin there is a nice introduction here:

And for coroutines in general check out this video:

Finally if you want to learn more about Actors implementation using coroutines in Kotlin, then head over here:

Kotlin/kotlinx.coroutines
_Library support for Kotlin coroutines . Contribute to Kotlin/kotlinx.coroutines development by creating an account on…_github.com

Thanks for reading! ❤️

_“aerial photography of woman on pink swimming floats” by [Unsplash](https://unsplash.com/@tom_grimbert?utm_source=medium&utm_medium=referral" rel="noopener" target="_blank" title="">Tom Grimbert on <a href="https://unsplash.com?utm_source=medium&utm_medium=referral" rel="noopener" target="blank" title=")