MeteorJS is a do-it-all framework for making JavaScript applications. If you enjoy making websites in HTML, CSS, and JavaScript, then you can use those skills to make apps for your PC or phone.

By default when you do “meteor create myapp & cd myapp & meteor run”, it serves an HTML/JavaScript web page along with a Node/MongoDB backend (which is unused at this moment).

Nodejs is simply the name for the JavaScript that sits on the server end. Mongodb is the NoSQL (not-only-structured-query-language) database that Meteor uses.

Let's start a mobile app demo

To get started, you type “meteor add-platform android” and then “meteor run android”. This will run this app on your plugged in phone (or virtual device) using your computer as the server (if you made something in the backend). You can do the same thing with an iPhone using a Mac.

The JS, HTML, and CSS files are intuitively organized within the 'server' and 'client' directories. This is the MVC (model view controller) design pattern.

The Android and iOS mobile interfaces are handled by Apache Cordova. You won't see it in a basic webapp. However you should definitely be aware if you are using any phone hardware functions.

The basic platform allows you to add on whatever other framework you wish to the back end or front end. Everything from Angular, Express, React and Vue can be installed on top of Meteor.

Popular CSS frameworks like Material-UI are usually used to ease the design work. However you don’t need to add any other framework at all. Meteor comes with a great Publish/Subscribe method, Blaze handlebars and user-accounts, and much more.

Beyond the Demo

Besides the plugins available with "meteor add", you also have access to all npm and cordova plugins. You can use "meteor npm install" to access them.

You can even add a desktop platform using Meteor-desktop. This uses the Electron framework. You can then make Windows, Mac, and Linux applications. Hopefully this functionality will be natively supported in Meteor version 2.0.

There has been a healthy community of Meteor developers in different forums since 2012. The documentation at guide.meteor.com is more extensive and clear compared to most frameworks.

Although this may seem to be the perfect shortcut for a new developer, I will warn you: Don't include a package or framework in your project until you are confident you know how it works.

Meteor is good at integration but it can take extra work to combine different packages. Don't just shop around for a list of packages expecting it will work all together perfectly.

Meteor is a great tool for a beginner looking to be introduced to the broad scope of app development and the process of building a simple app.

Production

Meteor can, of course, create full production web and mobile apps. It's used by several mid-size and large companies such as Ikea and Workpop.

For easy development and optimization, you can use Galaxy hosting. Galaxy will help get you to production without any system administration knowledge required.

If you do have knowledge and time, then you can host it on your own server/VPC. For example, a $5 a month AWS Lightsail instance can host an app with a hundred users.

Self-hosting and building works much the same way as you began the Meteor demo. However, instead of "meteor run" you will be building (meteor build) – your backend into a standard nodeJS app, and your mobile into a signed APK or IOS app.

There have been rumors over the years that Meteor doesn't scale well. This has been largely disproved and can be overcome with various techniques.

Meteor security has the typical high standards of a well maintained open-source project. Follow the security guide closely and watch out for noSQL injection.

Advantages of Meteor

• A diverse community of contributors gives the framework special resilience and longevity. Most other frameworks are created by a single mega tech company. This could mean that the project will get shelved if they see no return on investment. With Meteor, the direction of its development closely follows its users.
• It's cross-platform. Googles Flutter isn't going to work on Apple's iPhone. Meteor allows you to make all the implementations of your app in one place.
• It's built in MongoDB handlers and there's support for GraphQL.

Disadvantages of Meteor

• If developers place too much reliance on different pre-built packages they can conflict with one another.
• If you are just making a webapp it might be simpler to use Express.
• You can't make a mobile web-app run as efficiently as with native.

In conclusion, Meteor is an effective framework that can help you cut development time and ease app maintenance.

If you are looking to learn more on how to make Apps in JavaScript, read this new book from Manning Publications.

2020, still looking for a productive JS framework

When you create a product for your own company, you are free to spend time setting up a development environment that fits your own quirks. Granted, you'll likely spend a reasonable amount of time.

But when developing for others, you don’t have this freedom. You have to deliver high quality code in a predictable amount of time.

To be competitive, you have to scale across clients. Each app can’t be your first app. Knowledge, and generic code, must be reused. Most of the time that means relying on frameworks.

As an agency owner, I’ve always loved Meteor. It’s one of the rare JavaScript frameworks that's truly productivity-focused: a package-first architecture, an isomorphic approach, a persistence solution out-of-the-box…

I am also a core contributor of Sacha Greif’s Vulcan.js framework. Vulcan is an opinionated full-stack framework, implemented as a super-set of Meteor. It goes one step further by providing declarative patterns for very fast development and relying on Apollo GraphQL.

Everything (GraphQL schema, API, database structure, forms, data tables, and so on) is automatically generated based on a JavaScript schema. Cool, isn’t it?

But Meteor's limitations are a glass ceiling. I’ve had successful projects with Meteor and Vulcan, but I could never push those frameworks to bigger clients. Too many scalability issues, lack of traction, poor test tooling, you name it.

Back to square one, I needed to find a framework I could get married with.

Next vs Meteor?

That’s comparing apples and oranges!

When I first discovered Next back in 2017, it was a promising front-end-only framework. Front-end-only. I used it to build my company website and then forgot about it.

And then, people around me started to act weird. They suddenly talked about two frameworks with nothing in common, Meteor and Next, as if they were swappable. You traded Meteor for Next? Why not replace Express by Create React App while you’re at it?

You can even use create-react-app to fight hackers. Screenshot from the French TV show "Bureau des légendes" (Canal +).

As far as I remember, Reaction Commerce has been one of the first non trivial frameworks to do the switch.

I wasn’t convinced. And indeed, they still had to implement a GraphQL API on top of their Meteor app to communicate with their Next front end. Trading one framework for 2 is not the best bargain, so we kept Vulcan a Meteor-based framework.

If you think that comparing Meteor to Next is comparing apples and oranges, you are not that wrong. To this day, Next's catchphrase is still “The React framework”. Not “The Node framework”.

And yet, I started to change my mind a few months ago.

From front end to full stack with API routes

API routes were officially introduced in July 2019 with the v9 release.

That’s what got me on the Next bandwagon again. API routes mean that Next.js is now a minimalistic, but perfectly viable full-stack framework.

Vulcan is built around GraphQL. And GraphQL is a great fit for API routes. The API is served through a unique, dynamic /graphQL endpoint. In Next, this translates by creating a graphql.js API route. Easy peasy.

Nowadays, Next covers the full spectrum from static to full stack

Next is more and more referred to as a “hybrid framework”. That makes sense, as its versatility is extreme.

• You can develop a full-stack application with a serverless-style back end.
• You can develop a SaaS application with dynamic server-side rendering.
• You don’t want to maintain a server? You can follow the JAMstack philosophy, and export a static app with build-time server-side rendering.
• If you are allergic to client-side JS, you can go as far as removing JavaScript from the bundle and keep only HTML code.

But don’t think Next is a jack of all trades. It’s a serious contender with Gatsby in its static form. It’s a promising alternative to Meteor in its full-stack form. Vercel (ex Zeit) has been doing a tremendous job at keeping it both high quality and very lightweight, whatever the use case.

Don’t drop Meteor yet

I want to make something clear. Replacing Meteor by Next as our core framework in Vulcan does not mean we think Meteor should be dropped altogether.

There is one big thing we learned from our experiments with Apollo and Meteor in Vulcan: if you plan to use Meteor, just embrace the way it works. Forget about GraphQL. Forget about Webpack. Though created by the same people, Apollo and Meteor conflict a lot. It feels weird for GraphQL developers, it feels weird for Meteor purists.

Use DDP, methods, pub/sub, learn how to scale your app, join the forum, buy coffee mugs with Meteor’s logo on them. Now that Tiny has revived Meteor, it’s a safe bet for the years to come.

With Vulcan + Next, we simply strive to provide a GraphQL alternative to Meteor. It’s not worst or better, it’s the same philosophy with a different implementation.

A framework is like a music instrument. Don’t pick the trendiest, pick the one that fits you. If your instrument is Meteor, go for it.

Introducing Vulcan Next Starter, a state-of-the-art Next application

Using Next out of the box is perfectly fine. You get a build system, a styling solution, a rational folder structure.

But if you want to build an app for the next billion dollar startup, you’ll probably need a few more tools. Remember, we are seeking for a productivity-first alternative to Meteor.

A cool setup could be this one:

• Cypress and Jest for unit and e2e testing
• Storybook for visual testing and design documentation
• Internationalization, alias i18n (especially if you are from France like me :))
• TypeScript, to express your domain model through static types
• Material UI to get a solid set of customizable core components
• Apollo Client to communicate with GraphQL APIs
• Optionally, Apollo Server to setup a GraphQL entry point, with Playground and Voyager for API exploration

Next provides a handful examples in its core repo. But that’s not enough in a real life context. These tools can interact altogether in unexpected way.

Typing client-side only components like Leaflet or Plotly may prove difficult. Same goes for unifying the build system of Next, Jest and Storybook, or avoiding bad interactions between Apollo and Material UI during server-side rendering. Redirection with SSR means handling server and client scenarios jointly. I18n is especially hard to set up on its own. And the list goes on.

Trust me, you don’t want to tackle such issues alone. And guess what? We’ve gone through the hassle for you!

All those tools are installed in our new, shiny, Vulcan Next Starter.

We still have a long road ahead, but we are proud to say that it’s safe to use in production.

Next steps

Our boilerplate fulfills only half our promise. You get a cool front-end setup, but still miss a database and guidelines to implement the back end. That’s not really comparable to Meteor at this point. A handful of lambdas is not a framework. Neither is subscribing to cloud-hosted solutions.

That’s where Vulcan comes in. Through the years, we have crafted a powerful GraphQL API generator, with Mongo as the database and tons of front-end utilities. Naturally, the next step for us is to combine both Vulcan and Next to create a true full-stack framework.

You can follow our progress by joining us on Vulcan’s Slack.

Next and Meteor will be first-class citizens of Vulcan, but any kind of JS front-end or back-end technology could enjoy it. Be it Gatsby or a custom Node micro-service. At Vulcan, we sell apples AND oranges, as long as they make you an efficient developer.

Now, it’s time for us to get back to work, we have a lot to do. Hope to see you at Vulcan!

A starter from the trenches

Special thanks to Aplines, who trusted my company (LBKE) in using the latest technologies for their product. Thanks to them, we’ve tested all features included in Vulcan Next Starter altogether in a real-life professional application.

They are looking for developers, so if you want to learn more about using Next and GraphQL at scale, that’s the place to go: job@aplines.com

Meteor appeared at the beginning of 2012, rocking the web development world. The possibility of bridging the gap between the server and client sides of a particular website or web app was genuinely tempting.

Many experts thought that this capability alone should have turned the platform into a mainstream industry standard. Nevertheless, eight years later, many people claim that Meteor is dead.

Is that so? And is there any rational justification for learning Meteor? This article will give you a definitive answer.

Meteor is Dead! Long Live Meteor!

Many developers believe that Meteor is dead. The popular explanation is simple: being introduced in 2012, it already had a promising set of features in 2015 but failed to extend them significantly.

Some of you might have even heard of the problems with the funding of the development team, too. For many people, this fact alone is enough to forget about the platform once and for all.

However, the reality is much more optimistic than it may seem. Today, the developers of Meteor receive stable funding from Tiny, one of the most reliable investment funds in IT. The version history also shows that the development platform is far from being dead, with the most recent iterations having a proud 1.10.1 designation released in February 2020.

Thus, if you have always secretly liked Meteor but were afraid to choose the platform due to the constant reports of it being dead, now is the perfect opportunity to hop onto the hype train.

The reports of Meteor's death are greatly exaggerated, and the possibility of its success is greatly underestimated.

Unprecedented Simplicity: The Key Features of Meteor

Now that we have clarified the status of Meteor, it is time to describe some of its killer features. Meteor is undoubtedly among the most feature-rich and, at the same time, easy-to-use JavaScript frameworks today.

Killer Feature 1: Unified Client and Server Development

Meteor, as we have mentioned before, is notable for its ability to bridge the gap between the server and client sides of any project. Thus, every aspect of your website can be developed solely via JavaScript.

The benefits of this feature are boundless in the modern development world, and are valuable for both fledgling and the experienced developers.

Above all, this approach allows developers to create projects without experience with other programming languages.

Thus, a talented but inexperienced JavaScript programmer can easily maintain several Meteor projects without any problems. Experienced developers can go even further, maintaining whole ecosystems of services and products via the Meteor platform.

Consequently, the adoption of Meteor in your company can be an extraordinary boon for all involved parties.

• Your programmers will be able to develop more and spend less time on the exhausting process of learning additional frameworks and programming languages.
• Your customers will enjoy lower pricing on services.
• And you will reap higher profits after the adoption of the framework.

Independent developers can also be the major benefactors of the development platform today. After all, the ability to use only one language for personal programs will give you an opportunity to take on more freelance projects.

Killer Feature 2: Transforming Web Applications into Smartphone Programs

Every company that develops a specific web application seeks to have a smartphone version of its product. The benefits of such a strategy are rather obvious: after all, everyone has an Android or iOS phone today.

However, the businesses that seek smartphone versions of their apps often face the problem of massive development costs. The mobile versions of their web apps are often recreated from scratch.

If they are lucky, this process might only involve the client-side platforms. However, smartphone development frameworks are so divergent that sometimes both the client and the server have to be extended.

We are quite sure that you have seen frustrating situations in which certain web apps only have iOS or Android clients for smartphones. The lack of unity regarding frameworks is the main culprit in this case.

Meteor offers an elegant and extensible solution to this problem. Due to the potent integration of Meteor with Apache Cordova, you can quickly turn your web application into a smartphone app without any significant investment.

On the purely technical side, such a transition is possible due to the embedded container capabilities of Meteor and Cordova. All you would have to do is insert your web app into a pre-developed smartphone container.

Another critical design choice is the approach Meteor takes with data. Meteor uses data on the wire – the server doesn't send HTML but data, which is then rendered by the client. If the design of your web app is already touchscreen-friendly, you can immediately push your new program into the App Store or Google Play Market with Meteor.

If that is not the case, all you would have to do is slightly adjust the design using the JavaScript language. Essentially, the process would be similar to developing a mobile version of a website.

Once again, such an approach will save you unprecedented amounts of time and money. Instead of having to hire some dedicated smartphone developers, you will be able to fully concentrate on your web apps. This feature is also vital from a purely aesthetic standpoint. Meteor is one of the best tools when it comes to making the design of your products uniform.

The use of a unified platform will help you establish a professional standardized look in all business spheres.

Killer Feature 3: Real-Time Updates

Meteor is also capable of real-time updates, or so-called "full-stack reactivity." The changes you make immediately appear across all databases and style templates. In this way, you would be immediately able to see critical bugs and double check the features without having to tediously update web pages and certain programs.

This feature is of vital importance when it comes to large teams. The immediate updates are visible to all team members, creating a perfect environment for collaborative development. Ultimately, your web apps and smartphone programs will significantly benefit from this feature as it makes bug fixing incredibly simple.

Without going into details, one way of implementing the feature is a publication/subscription functionality.

// Code on the server side
const MyAwesomeData = new Mongo.Collection('myAwesomeData');
Meteor.publish('myAwesomeData', () => {
return MyAwesomeData.find() 
})

It creates a publication for everything in the collection myAwesomeData. This publish function is requested whenever a client subscribes to it. So, let's create a subscription.

// Code on the client side
Meteor.subscribe('myAwesomeData')

Now, all the subscribers will receive updates whenever a publication is requested. Also, we can receive data using a specific parameter.

// Code on the server side
Meteor.publish('myAwesomeData', (userName) => {
return Comments.find({ userName: userName })
})

The last piece of code retrieves data using a specific user name:

// Code on the client side
const userName = 'Jack Sparrow'
Meteor.subscribe('myAwesomeData', userName)

Killer Feature 4: Easy Package Management

Often, the deployment of modified versions of development frameworks requires a significant expenditure of time and resources. Meteor, however, is extremely user-friendly in this regard, offering some of the best package management tools on the market.

Today, the community of the Meteor developers maintains a gargantuan database of extensions on the AtmosphereJs website. Some of the popular extensions include tools for embedding ReactJS and Vulcan Bootstrap.

To install an Atmosphere package, you simply run the following command meteor add nameOfThePackage:

meteor add react-meteor-data

And to delete a package:

meteor remove react-meteor-data

To import and start using it in code, you should use the "meteor/" prefix:

import { useTracker } from 'meteor/react-meteor-data';

More information can be found here: https://guide.meteor.com/using-atmosphere-packages.html#peer-npm-dependencies

This feature is especially impressive considering its simplicity. The installation process only requires a set of simple commands that almost any advanced Windows or Linux user should already be comfortable with.

Even if you are an absolute beginner, you should have absolutely no problems with setting up a basis for even the most complex programs.

Thus, any person can swiftly create app prototypes and make them feature-rich by using this robust library of extensions and the advanced package management tools provided by some of the Meteor developers and the active community that surrounds the framework.

Killer Feature 5: Extensive Learning Resources and Documentation

One of the key problems of many open-source projects is the complete lack of proper documentation. This problem led to the death of countless promising projects as outside developers were often forced to essentially reverse-engineer their features.

Do not worry – Meteor is unlikely to fall victim to this problem. After all, the website of the platform has a powerful set of tutorials for beginners and a whole subsection is dedicated to documentation.

We recommend these resources to both the beginners and the advanced users.

If you are new to web development, Meteor tutorials will help you create your first web apps (one of the highlights includes a Whatsapp clone).

If you are an advanced developer, the tutorials will quickly teach you the basics of Meteor. After that, you will just need to check the documentation from time to time to resolve some issues.

Killer Feature 6: Active Community

If you fail to find information on the Meteor website, you can always resort to the robust forums.

We have spoken extensively about the death of Meteor in the first section of this article. Few facts defy this claim as much as the community of Meteor users.

As the massive extension library of Meteor proves, the framework receives active support not only from its developers but also from the community.

At the time of Meteor's presumed death, the interest from developers regarding the platform was seeing unprecedented growth. Consequently, the framework now has an incredibly passionate user base which is always ready to troubleshoot your problems.

Many people (including myself) know that Meteor’s community is among the nicest on the Internet. You are unlikely to meet any hostility there, and many active members are eager to help newcomers.

Since Meteor developers have received a significant financial boost from the Tiny Investment Fund, this community is likely to grow even more. Thus, your adoption of Meteor is likely to be very smooth as countless developers will be be ready to assist you.

Info-graphic. The Popularity Levels of Meteor. Courtesy of BuiltWith Trends Service. Accurate as of April 2020. Link: [https://trends.builtwith.com/framework/Meteor](Infographic 1. The Popularity Levels of Meteor. Courtesy of BuiltWith Trends Service. Accurate as of April 2020. Link: https://trends.builtwith.com/framework/Meteor.).

Conclusion: So, Should You Learn Meteor?

The short answer is simple: yes, undoubtedly.

Knowledge of the Meteor platform is the perfect addition to any developer's portfolio.

• If you primarily develop with JavaScript, the platform will help you or your company cut time expenditures on creating separate client and server projects.
• If you are an independent developer who uses C# or Ruby, Meteor can be a perfect entry point into JavaScript development. It will allow you to create independent web app projects with one programming language.
• If you are an absolute beginner, the platform will help you create your first well-functioning apps after just a few weeks.
• If you want to establish a startup, Meteor can be a perfect starting point for quickly creating a powerful prototype of your future web app. More importantly, the tool is robust enough to make any transition to other platforms unnecessary after prototyping.

Essentially, with Meteor, only the sky is the limit. Modern development is becoming more and more web-centric. Meteor perfectly reflects this trend, offering an all-encompassing platform for web developers.

By using it, you will be able to create a seamless integration of web and mobile apps, both of which are the future of consumer-centric computing.

Do you have an idea for a JavaScript project?

My company KeenEthics is an early adopter of the Meteor framework and a well-established JavaScript company in general. In case you have a promising project in mind, feel free to request an estimate.

If you have enjoyed the article, you should continue with What Are the Advantages of Node.JS? and Angular vs React: What to Choose for Your App?

A year ago I wrote an article describing my first experiences scaling Meteor. In short, I created a popular fantasy football website using Meteor. At a certain point, my service started slowing down. The single server I had running the game could no longer handle the load. I was able to solve these early scaling issues by — among other things — adding additional servers.

Well, when last summer’s new season of football arrived, I once again ran into scaling issues. Adding more servers alone wouldn’t solve these problems. But I did manage to overcome them.

This article will explain things I learned this time around, broken down into six pieces of practical advice.

One thing that has changed since my last article is that the Meteor Development Group has finally released Galaxy, which gives you Meteor hosting at $29 per container per month. This doesn’t include database hosting, but you can use something like Compose or mLab for that. Alternatively, you can self-host the database on AWS or DigitalOcean. This will be cheaper, but will require more work on your part.

I myself use DigitalOcean for hosting. The site runs on $5/month, 512MB droplets with one Meteor instance running per droplet. I use Meteor Up (Mup) for deployment and Compose.io for database hosting.

Whether to go with DigitalOcean or Galaxy is up to you. Galaxy will do a bunch of stuff for you and will save you time. Going the DigitalOcean route will save you $24 per container per month. For a lot of companies going with Galaxy makes the most sense since developer salaries will be far more expensive. In any case, I’ll leave the business decisions up to you.

Moving on. There are a few things that really helped scale our Meteor app this summer. We had some bad days. It really wasn’t smooth sailing at times, but we got through it.

A summary of lessons learned

Here are the major lessons I learned from my year of scaling:

Lesson #1: MongoDB indexes are super important!

Lesson #2: Having too many Meteor instances is a problem!

Lesson #3: Don’t worry about scaling Nginx.

Lesson #4: Disconnect users when they’ve been away for a while

Lesson #5: Will Griggs is on fire

Lesson #6: Take a cue from how they scaled Pokemon Go

Let’s go through these one by one.

Lesson #1: MongoDB indexes are super important!

So this one was an amateur mistake. Every article on scaling Meteor (or MongoDB) tells you to use indexes. And I did! But one index was missing, and I got burned for it — really burned — on the most important night of the year for us.

Explaining indexes by way of example. If you have 10,000 player scores and want to find the highest score, in a regular case Mongo would have to go through all these scores to find the highest one. If you have an index on the score, then Mongo saves a copy of all the scores in either ascending or descending order, and will find the highest score in a fraction of the time. You can read more about indexes on the MongoDB website.

In a Meteor project, I recommend having one publications.js file that contains all your publications. Below each publication you should have code that creates the necessary index for each publication. The code to create an index in Meteor looks something this:

Meteor.startup(function () {    Teams._ensureIndex({ userId: 1 });});

The _id field has an index by default so you don’t need to worry about that.

Getting into the details. I use Compose.io for MongoDB hosting. They’ve been fine, and support has also been okay, but don’t listen to them when they think all your problems can be solved by adding more RAM. This isn’t true. It might work sometimes, but in my case it was just nonsense advice.

I use Kadira.io for performance monitoring. Every Meteor app should use Kadira and the basic package is great and free, so no reason not to. (Update: Kadira is currently still an obvious choice for Meteor apps, but the team behind Kadira recently moved away from Meteor, so beware of that for the future.)

In Kadira I was seeing graphs such as these:

Super slow response time from 21:48 till 22:08

At a certain point the PubSub and Methods Response time become ridiculously large. Anything above 1,000 ms to respond is problematic. Even a 500ms response time can be bad. But 10–20 seconds as an average response time for an hour straight basically means your users hate you and your app is barely working for them.

In general, when things are performing slowly, I just add more servers. And I did that here too, except this time, adding more servers just made things worse. Far, far worse. But we’ll get to that later.

So at this point, what you do is you scramble around Google and spam StackOverflow and the Meteor forums.

Eventually I landed upon this gem in the Kadira dashboards:

Database is taking forever to respond

From this we see that the database is taking forever to respond. Adding more Meteor instances is not going to help us here. We need to sort out Mongo.

Kadira was no good at showing me why the database was responding so slowly. Every publication and method was showing a very high database response time.

The answer came from visiting Compose.io at peak times. On the dashboard, you can have a look at the current ops (current operations) that are running at any given moment. I saw something like this (but far worse):

I had no idea what all this mumbo jumbo was, but you’ll see that each op has a _secsrunning field. In the image above it says 0 seconds for everything, which is great! But what I was seeing during peak time was 14 seconds, 9 seconds, 10 seconds… for the different operations that were going on! And it was all coming from the same query being made by my app.

I ran this query myself and it really did take something like 16 seconds to get a response! Not good! And running it with explain (as some on the Meteor forums suggested) showed that 180,000+ documents were being scanned! Here is one of the problematic queries:

Problematic query

Anyway… lo and behold, there’s no index set up for this query. I added the following indexes:

Meteor.startup(function () {    HeadToHeadMatches._ensureIndex({ team1Id: 1, gameweek: 1 });  HeadToHeadMatches._ensureIndex({ team2Id: 1, gameweek: 1 });});

After this the whole database starts acting quickly again. This one problematic query was slowing down the entire database!

UPDATE #1: based on Josh Owen’s comment, a better way to do add indexes is to use Collection.rawCollection and createIndex, but the above code will work for you till at least Meteor 1.4.2.

UPDATE #2: indexes are more complicated than I first thought having run into trouble with them again this week. You probably won’t be able to find all your queries that need indexes without looking through your logs.

You need to find all queries that are using COLLSCAN. This means the query is not using an index and to find the document, Mongo has to loop through the entire collection to check if the document you’re searching for exists.

If you’re using Compose.io and are on MongoDB classic then you’ll need to email support to find which queries are using COLLSCAN. If you’re on their MongoDB 3.2 plan then you should be able to find these queries in their dashboard.

Also, if you suspect a query is problematic, run the query with explain() and you’ll be able to see how may docs are being scanned. If nscanned is equal to the number of documents in the entire collection, you have a problem and need an index. One bad index can massively affect your entire database since it will lock it down for all queries.

Lesson #2: Having too many Meteor instances is a problem!

So once you learn how to scale to multiple instances, you hope it’s the end of all the scaling misery. Alas, this is not the case. And adding too many servers will hurt performance at a certain point.

This is because Mongo uses additional RAM for each connection to the database. I must have had around 60–70 instances connected to my database at some point, and Mongo did not like it, nor did I need that many. The Meteor instances weren’t the bottleneck for performance.

You can give Mongo more RAM of course, but just be wary of what happens when you keep adding more servers. You might be taking the load off of each Meteor instance, but you’re adding load to Mongo creating a new bottleneck.

Lesson #3: Don’t worry about scaling Nginx

One thing I was worried about this summer was that Nginx would be my bottleneck. This will rarely be the case. Nginx should be able to handle thousands of concurrent users without a problem.

I did speak to a company that was having troubles with Nginx a few months ago. They had to handle a couple of thousand concurrent connections. You can read this article for some more tips about optimising Nginx for high traffic loads.

Some highlights from the article that are worth using immediately:

Turn off access logs:

By default nginx will write every request to a file on disk for logging purposes, you can use this for statistics, security checks and such, however it comes at the cost of IO usage. If you don’t use access logs for anything you can simply just turn it off and avoid the disk writes.

Worker processes and connections:

Worker Processes
The worker process is the backbone of nginx, once the master has bound to to the required IP/ports it will spawn workers as the specified user and they’ll then handle all the work. Workers are not multi-threaded so they do not spread the per-connection across CPU cores. Thus it makes sense for us to run multiple workers, usually 1 worker per CPU core. For most work loads anything above 2–4 workers is overkill as nginx will hit other bottlenecks before the CPU becomes an issue and usually you’ll just have idle processes. If your nginx instances are CPU bound after 4 workers then hopefully you don’t need me to tell you.

An argument for more worker processes can be made when you’re dealing with situations where you have a lot of blocking disk IO. You will need to test your specific setup to check the waiting time on static files, and if it’s big then try to increase worker processes.

Worker Connections
Worker connections effectively limits how many connections each worker can maintain at a time. This directive is most likely designed to prevent run-away processes and in case your OS is configured to allow more than your hardware can handle. As nginx developer Valentine points out on the nginx mailing listnginx can close keep-alive connections if it hits the limit so we don’t have to worry about our keep-alive value here. Instead we’re concerned with the amount of currently active connections that nginx is handling. The formula for maximum number of connections we can handle then becomes:

worker_processes worker_connections (K / average $request_time)

Where K is the amount of currently active connections. Additionally, for the value K, we also have to consider reverse proxying which will open up an additional connection to your backend.

In the default configuration file the worker_connections directive is set to 1024, if we consider that browsers normally open up 2 connections for pipe lining site assets then that leaves us with a maximum of 512 users handled simultaneously. With proxying this is even lower, though, your backend hopefully responds fairly quickly to free to the connection.

All things considered about worker connections it should be fairly clear that if you grow in traffic you’ll want to eventually increase the amount of connections each worker can do. 2048 should do for most people but honestly, if you have this kind of traffic you should not have any doubt how high you need this number to be.

Lesson #4: Disconnect idle users

This one is important! I don’t why this isn’t a bigger thing in the Meteor community!

Disconnect users when they’ve just left their tab open. It’s so simple to do and saves precious resources.

To disconnect automatically you can use this package: mixmax:smart-disconnect.

Lesson #5: Will Griggs is on fire

If you got this far in the post, you’re probably feeling super inspired and in the mood for a football chant. I present you with Will Griggs:

There wasn’t actually a point here. It just seemed like the appropriate thing to write at this point in the article. But if we actually want to learn a lesson from it then here goes:

If you’re a solo developer, and you have thousands of people relying on your app to work right now, things can get stressful. My advice to you (and to myself): calm the heck down. Listen to some Will Griggs on Fire. Hopefully you’ll work it out, and even if things mess up, it’s probably not as bad as you think.

Pokemon Go was pretty awful at the start. Servers were constantly overloaded, but people kept coming back to play. From a business perspective, Niantic still made a killing. The hype has now died down, but that has nothing to do with their scaling issues, or the many early bugs. It’s just the end of the fad.

So the life lesson, listen to Will Griggs when you’re stressed out.

Lesson #6: Take a cue from how they scaled Pokemon Go

On the topic of Pokemon Go, lets talk a bit about what happened. Firstly, Pokemon Go will not happen to you. Pokemon Go had a strong team of ex-Googlers that knew how to do deal with enormous loads, but even they got caught out with the popularity of the app. They were ready for a big load, but not or a load the size of Twitter.

Some apps around Pokemon Go also popped up. Pokemon Go chat apps, and Pokemon Go Instagrams started popping up and became very popular, very quickly with a million users in a matter of days. Some of these apps were developed by solo developers and handling the load was a challenge for them.

There’s this article about how someone built a Pokemon Go Instagram app with 500,000 users in 5 days and ran it on a $100 per month server. That’s impressive. And the takeaway from the article is that you can build a quick MVP that scales if you know what you’re doing.

If you can do that, that’s definitely great, but if you’re a young and inexperienced developer that may not be possible. I would recommend to go and build your dream app and not worry too much about what happens when you need to scale.

If you can build things the right way from the get go that’s definitely a plus and it’s definitely worth asking more experienced developers for advice to try and get things right from the start. But don’t let the fear of scaling hold you back from creating your dream app. The harsh reality is that people probably won’t like your app and it would be impressive if you can get 100 people to use it.

But following the principles of the lean startup, it’s better to build something, get it into the hands of real users, and get feedback, than to never launch due to fear of not being able to deal with a heavy load.

Looking Ahead

These episodes dealing with scaling have been a burden and ideally I would have preferred not to have to deal with these issues. It would be great if things just worked and you could push off scaling issues as long as possible. Because of this I’ve started looking at other platforms that handle scale better.

One such platform is Elixir which is built on Erlang. Erlang is what Whatsapp uses and allowed a team of 35 engineers to scale to 450 million users! Even today, with close to 1 billion users, Whatsapp has a team of only 50 engineers! That’s pretty incredible and you can read more here. How did they achieve such awesome scale for a real time app with so few people? The answer is Erlang. And today you can utilize the power of Erlang with Phoenix Framework and Elixir. We’re still using Meteor, but some aspects of the app I am considering moving over to Elixir which will enable us to hit large scale live updates.

I’d also take a look at Apollo which will work with Meteor or any Node.js server. Apollo will help you scale Meteor, because you don’t need every single publication to be reactive when using Apollo (which are the biggest drain on server CPU for Meteor apps.) You can achieve a similar result using Meteor methods to send data instead of publications.

One last point is that despite many influential Meteor developers recently leaving the community, there have been some developments on the scaling front with regards to scaling. Check out the redis-oplog package and discussion for more. It’s a very new package and I’d still say it’s in beta from my little experience playing with it a week ago.

If you enjoyed this post, give it a heart, and if you want keep up with the latest in inspirational scaling articles, give me a follow.

I recently went through the challenge and ordeal of having to scale my Meteor app. It’s a project that had already been running in production for about a year. This summer the app became a lot more popular with thousands of preseason signups. My initial setup could no longer handle the load and I was faced with a scaling problem that had to be solved quickly.

This article describes the process I went through and some of the things I learnt along the way and my hope is that it will help others that face similar challenges in the future. It will cover the basics, such as what scaling is, and how load balancing works. It will also walk you through some basic setups, and show you how to scale your Meteor app.

Before the summer I didn’t have any prior experience scaling apps, and despite the fact that I had read up about the topic quite a bit, when I had to actually sit down and start dealing with the challenges myself I felt quite lost and wasn’t sure I’d be able to fix the problems. I also felt that many of the articles I had read online assumed a fair amount of knowledge that I didn’t yet have.

My hope is that this article will help bridge some of that gap. I’m far from an expert on scaling Meteor applications, but I hope this article provides value to others that find themselves in a similar situation to the one I found myself in.

Background

A bit of background about the app under discussion. It’s a draft-style fantasy football game for the English Premier League (soccer). Most of the signups happen the month before the season starts. Apart from all the signups, nearly all the drafts also happen in this month. The draft is when all users in a league come online to pick their team. Users pick their football players one at a time and the entire process is live with each user having between 30 seconds and 5 minutes per player pick.

At peak times the site had over 500 concurrent users online and 20 simultaneously drafts running. If the server becomes unresponsive during a draft, it completely ruins the user’s experience with players being picked for him automatically that he didn’t necessarily intend on picking, so it was extremely important to avoid this.

So What Is Scaling?

To many this might seem obvious, but not so long ago, I had no idea what this buzzword meant.

Scaling is what happens when your app takes off. Your server can only handle so much load. What happens when 10,000 people want to use your app at the same time? Your server won’t be able to handle it, so you either need to get a better server, or get more servers to run the app. This process is called scaling.

There are two ways to scale an app. One is known as vertical scaling, and the other is known as horizontal scaling. Vertical scaling involves getting a more powerful server. Horizontal scaling involves serving your site from multiple servers.

To use an analogy, if you have a shop and employ one worker and want to be able to serve more customers, you can either get a better, faster worker (vertical scaling), or you can hire more employees (horizontal scaling).

For web apps, vertical scaling is usually easier to do. At hosting companies such as DigitalOcean or AWS (Amazon Web Services), you can easily upgrade your virtual private server to a stronger configuration with increased RAM, CPUs and storage.

The problem with this approach is that there’s always a limit to how powerful you can make your server. At some point you’re going to have to scale horizontally. Using the example above, your business can only go so far with a single employee. At a certain point you’re going to have to hire more staff members.

When it comes to Meteor apps, you’re going to have to scale horizontally fairly early on. A Meteor app is really just a NodeJS app and therefore runs in a single process which means it can only make use of a single processor.

We can make use of multiple processors by running multiple instances of our app at the same time. These processors could belong to the same server or spread across multiple different servers. Running multiple instances of an app means running the same app on different IPs or ports, and spreading the load over the different instances of the app. All instances of the app still connect to the same database and all connected clients will instantly receive any database updates irrespective of the app instance they’re connected to (assuming you’re using MongoDB’s Oplog tailing feature. Otherwise the updates might take a few seconds).

Whether we run multiple instances of our app on a single server with multiple cores, or run multiple instances on multiple servers, what we’ve done is horizontal scaling and things work the same way in both cases.

Each platform will have its own scaling challenges. This article is about scaling Meteor apps. On other platforms you’ll be able to get away with vertical scaling for far longer. StackOverflow runs on 25 servers and could even make do with 5. That’s some serious load on each server and is mostly a case of awesome vertical scaling.

The Different Parts Of A Meteor App

We can split our app up into two logical components. The first part is the server that handles users’ requests sending the appropriate data back, and performs any tasks that need to be performed. The second part is the database that stores the app’s data.

The server interacts with the database querying it and updating it on a consistent basis.

We’re not going to talk too much about scaling MongoDB. It’s probably going to be a while before you reach a point at which you’ll have to scale your database. If you’d like to know more about this topic, this link could be a good place to start.

Things To Do Before Scaling Up The Server

A good place to start before you get down to scaling is to make sure your app is running as efficiently as possible. The recommended tool to use to check the performance of your app and which methods and publications are taking the most time to complete is Kadira. The basic plan is free so there’s really no reason not to use it. Use the articles at Kadira Academy to work out how best to optimize your app and where to devote your energy. In general, you want to optimize the methods and publications that run most often and take a long time to complete.

One thing that is absolutely necessary for good performance is to make use of MongoDB indexes. You do this in Meteor by writing:

_Posts.ensureIndex({userId: 1});

This creates an index on the userId field in the Posts collection. You can also create indexes in MongoDB itself. See more on MongoDB indexes here.

See this article by Differential for some additional performance tips.

When Do I Need To Start Worrying About Scaling?

If your Meteor app has over 100 users online at any one time, you’re probably going to have to start worrying about scaling. (Of course, you don’t know when your app is going to take off. You could jump from 5 to 500 concurrent users in a single day, so it’s worth being prepared before this happens.)

Depending on your app, you might be able to get up to 500 concurrent users on a single DigitalOcean droplet. My app struggles with 100–150 concurrent connections, at which point it hits 100% CPU. The bottleneck for most Meteor apps seems to be CPU and not RAM, so scaling horizontally is a must. You can add all the RAM you want, but it won’t help your app. Lack of CPU is what’s going to overload your app and you can only get more CPU power by making use of multiple CPUs (or multiple servers).

The following graph from Kadira shows what happens when your app is under too much pressure.

Kadira stats. 1 core, $10/month Digital Ocean droplet

At around 7:28 PM you can see that the average response time for publications was about 18 seconds and for methods about 7 seconds. This isn’t a good situation to be in. The high response time was caused by a big spike in CPU which hit 100% for a couple of minutes. RAM isn’t an issue because only about 500MB is being used at any one time and the VPS being used here has 1GB RAM.

Deployment

There are multiple ways to deploy your Meteor app. Here are some of the common solutions people use for production apps:

• Self-hosted on DigitalOcean or AWS
• Modulus.io

There’s also the free meteor.com hosting, but this should only be used for development purposes. MDG also just released Galaxy, but the pricing for that currently starts at $500 a month. MDG is working on providing cheaper plans for Galaxy as well as a free plan, but at the time of the writing of this article, these aren’t an option.

It’s also common for people to host the database at another provider. A popular one is Compose.io.

Update: You can now get Galaxy hosting starting at $29 per month. Compose.io no longer has a free plan. Another Mongo hosting provider called MLabs does. When you go into production (i.e. when development is over and people are actually using your app), you do not want to be using a free plan however.

My own setup is DigitalOcean + Compose.io.

If you’d like to read about other setups, see the deployment section on Meteorpedia.

My initial setup was a $10 per month DigitalOcean droplet. This gives you a one core VPS with 1GB RAM. You also get 30GB of storage, though you probably won’t use most of it.

I use Compose.io’s $18 per month Elastic Deployment plan for MongoDB database hosting. You can deal with the MongoDB stuff yourself, but it’s just extra work that I didn’t want to deal with.

To deploy I use a tool called Meteor Up (or Mup for short). You can take a look at it on GitHub here.

If you deploy on Modulus.io, they’ll take care of a lot of the scaling issues for you. If you want to run multiple instances of your app, all you have to do is move a slider up and down on the Modulus website. The more instances you run, the more it will cost you, but you can scale down whenever you want too.

I didn’t go with Modulus for deployment because I had some problems setting it up two years ago. I assume these issues would have been ironed out by now. The other issue is cost. It’s cheaper to deploy on DigitalOcean. The downside is that it might take up more of your time. Another advantage of going with DigitalOcean is finer grained control over your server.

An article comparing DO, Modulus.io and Heroku can be found here.

So how do you actually scale a Meteor app?

As noted above, my app is deployed on Digital Ocean using Meteor Up. With this setup you don’t have the luxury of using a slider to load up more instances. A standard mup.json file looks something like this:

{   “servers”: [     {      “host”: “123.45.678.901”,      “username”: “root”,      “pem”: “~/.ssh/id_rsa”,      “env”: {}     },     {       “host”: “333.22.444.555”,       “username”: “root”,       “pem”: “~/.ssh/id_rsa”,       “env”: {}     }   ],   “setupMongo”: false,   “setupNode”: true,   “nodeVersion”: “0.10.40”,   “setupPhantom”: true,   “appName”: “myapp”,   “app”: “/Users/arunoda/Meteor/my-app”,   “env”: {      “PORT”: 80,      “ROOT_URL”: “http://myapp.com”,      “MONGO_URL”: “mongodb://arunoda:fd8dsjsfh7@hanso.mongohq.com:10023/MyApp",      "MONGO_OPLOG_URL": "mongodb://.....",      “MAIL_URL”: “smtp://postmaster%40myapp.mailgun.org:adj87sjhd7s@smtp.mailgun.org:587/”   },   “deployCheckWaitTime”: 15}

Notice that I’ve listed two servers in the servers block. When you’re first getting started, you’ll only have one server listed. With the following setup, we’ll be deploying to ip addresses 123.45.678.901 and 333.22.444.555. After deployment, if you were to visit either of these ip addresses in your browser, you’d see the same thing and both servers would be connected to the same database.

Load Balancing

So we’ve deployed our site, but we don’t want users visiting the site at random ip addresses. We want them visiting our domain. So let’s say our domain name is awesomedomain.com (which is better than example.com, in my humble opinion). How do we get each of our two servers to deal with half of the requests?

One way of doing this is by using a tool called Nginx. From Wikipedia:

Nginx (pronounced “engine x”) is a web server with a strong focus on high concurrency, performance and low memory usage. It can also act as a reverse proxy server for HTTP, HTTPS, SMTP, POP3, and IMAP protocols, as well as a load balancer and an HTTP cache.

We’re going to use it for three things:

1. As a reverse proxy
2. As a load balancer
3. For SSL support

What this means is that we’re going to run Nginx on one of our servers on port 80. Any traffic coming into the server from the web will be received by Nginx. Nginx will then forward the traffic on to instances of our Meteor app, which could be running on the same server on a different port, or a different server. Nginx will try and balance the number of requests that are sent to each instance.

Nginx being a reverse proxy means it forwards requests on to other places and then responds to the user.

Being a load balancer means it will split (balance) the load between the different instances of the app.

We’ll also use Nginx to provide us with SSL support. SSL support means that any data transferred between the user and our servers will be encrypted. Without SSL support, our Meteor site won’t work correctly in many countries, and it will also be fairly easy for people to hijack our users’ accounts or read any data that is transferred. Any site that uses HTTPS has SSL support.

SSL isn’t the focus of this article, but it is a must for any production app and Nginx can take care of the SSL stuff for you. To set up SSL with Nginx, see here. You don’t need to use Nginx for SSL support though. Meteor Up can also handle it using a tool called stud as shown here.

So how do we setup Nginx?

To do this, I recommend following the first step of this tutorial:

How To Deploy a Meteor.js Application on Ubuntu 14.04 with Nginx | DigitalOcean
_Deploy a Meteor.js application on Ubuntu 14.04 with Nginx and MongoDB. This tutorial shows you how to build and deploy…_www.digitalocean.com

Don’t bother following the rest of the tutorial. Meteor Up will take care of all that stuff for you automatically.

The article also explains how to use Nginx for SSL support.

Once all of that is working for you, you should have your app running at some domain. So far, Nginx hasn’t actually done much for us (except possibly SSL support.) We’re still not making use of our second Meteor instance, but that we’re going to fix that now.

To make use of more servers, you need to add the following code to the top of your Nginx file:

upstream myAppName {  ip_hash;               # for sticky sessions, more below  server 123.45.678.901:3000;  # server 1  server 333.22.444.555:3000;  # server 2}

Your Nginx file should now look something similar to this:

upstream myAppName {  ip_hash;               # for sticky sessions, more below  server 123.45.678.901:3000;  # server 1  server 333.22.444.555:3000;  # server 2}server {  listen 80;  server_name www.myapp.com  # and all other "server" directives  location / {    # the "hostname" below must be same myAppName from upstream directive above    proxy_pass http://myAppName;    # and all other "location" directives  }}

…where myAppName would be something like example.com or app.example.com.

The above code was adapted from Meteorpedia.

If you now reload Nginx (as described in the Digital Ocean article above) using:

nginx -t # check everything is okaynginx -s reload

The load will be split among your two Meteor instances.

If you’d like to add more instances in the future, all you have to do is add another line to Nginx upstream block and restart Nginx. Here’s an example that uses 4 Meteor instances, with 2 instances running on two different servers:

upstream myAppName {  ip_hash;  server 10.0.0.1:3000;  # server 1, core 1  server 10.0.0.1:3001;  # server 1, core 2  server 10.0.0.2:3000;  # server 2, core 1  server 10.0.0.2:3001;  # server 2, core 2  # or whatever other appropriate combination}

Removing instances just involves removing lines from the upstream block and restarting Nginx.

Load Balancing Algorithms

So if you just want to get things working, then following the above steps should work, but if you want to understand a little deeper, here’s a basic explanation of what we just did.

We’re using Nginx as a load balancer. For each request that comes in, Nginx has to decide which server to send it to. At the same time, Nginx has to be as quick as possible in doing its job, using as few resources as possible. Nginx is capable of handling 10,000 concurrent connections and simple load balancing algorithms are part of what makes this possible.

A simple algorithm that is used to decide where to send the next request is called “round robin.” In this algorithm, the load balancer sends client requests to each server in turn and once it gets to the end of the server list, starts the process again. The outcome for three servers is: 1, 2, 3, 1, 2, 3, 1, 2,…

There are many other algorithms that can also be used to decide which server to send a request to, but one constraint that Meteor requires us to follow is that each request from a user should be sent to the same instance. This is also known as “sticky sessions.” A user sticks with the same server the entire session.

This means we can’t use the round robin algorithm for our Meteor app, which doesn’t take into account which server the user first connected to. Instead we use the _iphash algorithm which is explained in the Nginx docs as follows:

With ip-hash, the client’s IP address is used as a hashing key to determine what server in a server group should be selected for the client’s requests. This method ensures that the requests from the same client will always be directed to the same server except when this server is unavailable.

Running Locally

You can try out running two instances of your app locally too. To do this open up two terminal windows and start one Meteor app on the standard port (3000) by running:

meteor

The database will now be running on port 3001.

In a separate terminal window run another instance of the app on port 4000 that connects to the same database by running the following:

export MONGO_URL=mongodb://localhost:3001/meteor

meteor --port 4000

You can now visit localhost:3000 or localhost:4000 in your web browser and both will make changes to the same database.

Things to be Aware of

There might be code in your app that you only want running once. For example, in my fantasy football game, I only want one server updating scores.

Another issue that can come up is that certain operations can be quite expensive. Again, updating scores in the game is quite a long process and I don’t want it interfering with the basic functioning of the app.

So how do you solve these problems?

There are multiple ways to deal with such a situation, but I’ll just outline what I did.

I set up a new droplet on Digital Ocean that doesn’t receive any traffic from the main website. It can be accessed at its IP address, but this droplet doesn’t receive any traffic from the website’s public URL. This instance is in charge of running all background tasks and any task that should only run once.

To make code specific to only one instance of the app, I make use of environment variables. In your local environment you can export variables in your terminal window as follows:

export ENV_VAR=valueOfOurVar

We did this before for _MONGOURL.

You can access environment variables in your Meteor app using:

process.env.ENV_VAR

In my app I can do something like this if I only want one instance of the app to run certain code:

if (process.env.RUN_BACKGROUND_TASKS === ‘true’) {   // do some task}

I could also have used a pure microservices approach, but this would have taken a bit more time for me to set up and code sharing would have been harder.

When using Meteor Up, you can set environment variables for a specific instance as follows:

“servers”: [   {     “host”: “...”,     “username”: “...”,     “pem”: “...”,     “env”: {       “ENV_VAR”: “123”,       “RUN_BACKGROUND_TASKS”: “true”     }   } ],

There are also packages that don’t work well when running multiple instances of an app either. One package I came across that falls into this category is mizzao:user-status. This package tells you which users are online at any given time and doesn’t work properly when running multiple instances. One server will have an open connection to a specific client while the rest won’t and this package doesn’t know how to deal with such a situation. A closed connection implies a user is offline. A work around is to use the konecty:user-presence package instead.

You don’t want cron tasks running on multiple instances either. For this you can use the percolate:synced-cron package. I didn’t personally make use of it, but it looks like a good solution opting instead to restrict all cron tasks to a specific server, but it looks like a good solution.

Other Solutions

There are many ways to do what we did in this article. Using Nginx is not the only solution. For SSL support, we could have used stud instead as Meteor Up does. For load balancing we could have used a tool called HAProxy whose sole purpose is to be a load balancer.

Another approach to all this is to use the MeteorHacks Cluster package. It’s simply a Meteor package which you add as follows:

meteor add meteorhacks:cluster

To get see it working immediately on your own machine export the following environment variable to your terminal:

export CLUSTER_WORKERS_COUNT=2

If you now run your app, you should see two instances of your Meteor app start up. Here we made use of the cluster package’s multicore support. We can also use cluster to distribute load among multiple servers. You should look at the cluster docs to set this up.

Is it better to use MeteorHacks Cluster or Nginx?

I don’t know the answer to this question. I did have problems using MeteorHacks Cluster in production when making use of its multicore support. The app would work fine under low stress conditions, but under heavy load the app would randomly crash required a manual restart to get it working again. There’s an open issue on GitHub for it here. If you don’t use the multicore support this shouldn’t be an issue though.

My Final Setup

The database runs on Compose.io at a cost of $18/month per GB of space used.

The app currently runs on 8 DigitalOcean droplets:

($5/month droplet has 512MB RAM and 1 core, $10/month droplet has 1GB RAM and 1 core, $20/month droplet has 2GB RAM and 2 cores.)

• 1 $5/month droplet takes care of background tasks and doesn’t handle any user requests.
• 1 $20/month 2-core droplet runs Nginx and 2 Meteor instances of the app.
• 4 $10/month droplets (1 instance per droplet)
• 2 $5/month droplets (1 instance per droplet)

There’s also a $5/month droplet for the blog that runs on Ghost on a subdomain (this isn’t Meteor).

I’m also working on a mobile app and this will run on other droplets and will serve a different site to visitors. The client views/logic here is different, the server logic is the same.

Which Droplets To Choose?

So you may be wondering why my droplets are different sizes. At first I only used $10/month droplets, but since my bottleneck is CPU and not RAM, I should be able to replace all my $10/month droplets with $5/month droplets, since you get the same CPU power with each.

I’m in the process of testing this out now and seeing how it goes. You need to be careful that you don’t run out of RAM when using 512MB droplets. If you use the spiderable package, your server can crash when a search engine indexes your site and it will cause CPU to spike to 100%. You can get around this by using prerender.io instead of spiderable and/or adding swap space. If you’re using React then you can use server-side rendering for search engines instead of spiderable as explained here.

Autoscaling

It would be nice to be able to automatically load up more droplets when the site is under stress and take them down when no one is using it. My site is used a lot more during the weekends for example. It would be nice if I could automatically launch more droplets whenever there’s a lot of traffic and take them down when there’s less traffic.

I haven’t seen an easy solution to this yet for Meteor apps deployed on DigitalOcean, but making use of the DigitalOcean API this should be possible without too much work. It would cut down hosting costs significantly.

It seems to be quite easy to autoscale on Modulus as shown here. And you can read about autoscaling on AWS here and here.

Things I’d do Differently

I’d test that my app can run well on two servers at a much earlier stage in the development process. The jump from 1 server to 2 servers is fairly big if you’ve never done it before. The jump from 2 servers to 10 servers is tiny.

Other Resources on Scaling Meteor Apps

• meteorhacks:cluster
• Meteor Up (Mup)
• Meteorpedia
• MeteorHacks: Does Meteor Scale?
• Discover Meteor : Scaling Meteor: The Challenges of Real-time Apps
• Differential: Scaling Meteor to 20,000+ Users in 7 Days

Later Additions

I really liked the following post. It has some key info on load testing, a better Nginx setup, and using a CDN: https://medium.freecodecamp.com/minimum-viable-devops-919972dfd9e0