Building an app is a difficult task in itself. From conceptualization, to market research, design, and development, it takes a whole lot of resources.

But if you are making something for the public, you know building it is only half of the work, right? If you want your app to reach as many people as possible, you will have to take another step: marketing.

Marketing is another beast altogether. And, it needs as much commitment as building the app. It's something you need to tackle even in the earliest stages of app development.

But the problem that plagues most solo developers is, "Where do I get the money?"

Traditional marketing does require quite a bit of cash. Advertising is one of the most lucrative industries, after all. Still, there must be something the rest of us cash-strapped individuals can do, right?

Of course there is. If you are on a limited budget, here are a couple of tips to get you started with marketing your masterpiece of an app.

Be Active on Social Media

No matter what your app is about, most of your audience will be on social media. And leveraging social media is one of the most cost-effective ways to promote your app.

Seven out of ten American adults use some form of social media. Among them, the most popular platforms are YouTube (81%) and Facebook (71%).

You can sign up to all the basic social media platforms free of charge and craft your campaigns from there.

Social media makes it easier for potential users to find your app and see what it’s about. These platforms are an avenue for people to interact, giving you a direct line to your audience.

It is also a great avenue to answer questions and interact with users. When done well, social media marketing can build up your reputation. It can also generate word-of-mouth marketing.

Creative social media campaigns can generate a lot of buzz around your app. A lot of brands and apps have surged in popularity due to well-executed campaigns.

To launch a successful campaign, you will need some good market research. It's a good thing that most social media platforms will give you access to your account analytics.

It's a good practice to keep your eye on it, as your analytics provide a wealth of information. It shows which posts gain the most attention, your audience demographics, among others.

All these for an initial payment of $0. If you have the budget, you can transition to paid ads to target your desired audience and get more buzz.

Create a Website or Blog

Building a website is one of the most established ways to promote online. It gives users a one-stop-shop to get acquainted with your app and its many facets. A self-titled domain also gives your app more credibility.

You can add different pages describing your app and its features. Maintaining a blog is also an excellent way to provide updates and boost your search engine rankings.

But as most of us know, building and hosting a website can cost some money. If you're on a tight budget, don't worry, you still have some choices.

Many websites offer free website building and hosting, which can fit a variety of needs. Some popular examples are WordPress, Wix, and Weebly. Building a website on WordPress is as easy as 1,2,3, as evidenced by the screenshots below.

Their free services will add their brand to your site domain. So your web address will be appname.wordpress.com, for example. If you want your own domain name, you will need to pay a monthly fee.

Still, you can check out their website builders, which are pretty user-friendly. You can use their drag-and-drop features for a more convenient experience. But if you want, you can also access the code for more control.

Image source: unsplash.com

Reach Out to Influencers

With the popularity of social media, influencer marketing has seen a steady rise.
What are influencers, anyway? Influencers are social media users with a significant following. Due to their established audience, they become an attractive avenue for advertising.

A majority of marketing companies have a standalone content marketing budget. Seventy-five percent of them intend to have a dedicated budget for influencer marketing.

Most influencers have followers that trust and admire their lifestyle and choices. Partnering with an influencer is an excellent way to add social proof. It can also build your app's reputation.

There is a whole lot more to influencer marketing. You will need to take some thought when selecting who you want to promote your app.

Selecting influencers you're going to work with requires you to know your audience. Look for someone within the same demographic or someone who has the same interests.

More popular influencers often charge a more significant fee. So if you're on a budget, you might want to look at ones with a smaller yet still considerable following.

Working with smaller influencers or micro-influencers is not a bad thing. Micro-influencers are a popular choice among brands today.

Prominent influencers are now more like celebrities. Micro-influencers have fewer followers but often come across as more genuine. They can make organic content with a more robust audience connection.

People rely a lot on word of mouth when making purchases. They are more likely to follow the advice of someone they trust. Micro-influencers have this kind of relationship with their audience. Working with them can add a boost to your app.

Get Your App Featured

When it comes to promotion, nothing beats a feature from a trusted media outlet. It's one of the best ways to reach even more casual internet users.

This is where you and your team's PR skills come in. You will need to write pitches that you can send to websites where you want your app featured.

To get higher chances of approval, do some research about these websites first. What kind of content do they usually produce? What writing styles do they use? Tailor your pitch around these factors.

You can also reach out to official review sites and ask them to do a feature on your app. If you've already released your app, you could provide them with a download link. If it's still under development, you could offer demo videos and high-quality screenshots.

Some other ways to get featured are through podcasts, vlogs, or other media. Reaching out can be pretty time-consuming, so get your networking skills ready!

Work on Your App Store Optimization

On Google Play alone, there are already 3.48 million apps. How do you stand out from an ocean of competition?

App Store Optimization (ASO) is one of the things to consider when promoting your app. It is like Search Engine Optimization (SEO), but for app stores.

Your marketing campaigns may be top-notch. Still, without ASO, you're missing out on most of your audience. A significant 70% of users find apps via app store search. Furthermore, 65% of downloads happen right after a search.

What does this mean? Pay attention to your ASO. If you can, create a campaign around it, but here are some key factors you should pay attention to.

• App Name. It needs to be catchy and relevant to your app's features.
• App description. Make sure to incorporate relevant keywords to help your app rank higher.
• Icon. Create an eye-catching icon that will attract users.
• Screenshots and videos. Prepare short videos and attractive screenshots that show your app at its best.
• Ratings and reviews. User feedback matters. Many users look at reviews before downloading, so keep up your customer service!

One last thing for ASO: Do your research and look at the top apps on Google Play right now. How did they structure their app page to attract and sustain the most users?

You can use a nifty tracker from the similarweb Blog to look at the most downloaded apps on Google Play. Data is an excellent way to study the factors that affect an app's performance.

Final Words

We can never deny that funding makes a lot of things easier. But if you lack in this department, creativity can go a long way.

There are tons of ways to promote your app. Some of them cost money, but you can make up for it with some skill and hard work. Make do with what you have and can gain success from it.

The desired benefits of agile development are many. Customers are happier and more willing to buy. Lead time from idea to delivery is shorter. Employees are more motivated and more productive.

Sounds good, doesn't it? In practice, certain behaviors stand in the way of the benefits. As a scrum master and agile coach, I see the same anti-patterns in many companies over and over again. Today's anti-pattern is about project budgeting and contracts.

Fixing the scope - not a good idea

In many companies, the scope of delivery must be decided on first. Then someone estimates the effort and calculates the necessary budget. Finally, there is a go or no-go decision for the project.

It is similar for many customer supplier relationships. First, a contract must be set up that describes exactly what will be delivered. Otherwise there is no order.

How else could it be? How else can you, as a customer, be sure what will come out in the end?

As an agilist, here's my answer. Customers can't possibly know the exact end result in advance. Software development is full of surprises. The requirements of today are history tomorrow.

It's important to recognize the tension between predictability and changeability. You can't have both: perfect predictability of the end result at the beginning and maximum consideration of change requests later.

What you can do instead

At the beginning of agile development, it is sufficient for stakeholders to agree on a product vision, and maybe a rough roadmap. Funding can be done incrementally, sprint by sprint.

But what if this is not possible in your company? What if the stakeholders don't want to give up the illusion of safety that comes from clarifying requirements early?

Well, it's a waste of time to specify details of requirements that will change later. So what to do?

You can agree on user stories in advance. That makes sense. But don't define the acceptance criteria! Clarification of details is postponed to development, shortly before implementation.

The advantage: you can react to lessons learned from development and new customer needs.

Change management done right

Either at the beginning of a project or during contract negotiation, stakeholders should agree on the mode of cooperation and the handling of changes later.

It is important to follow a few important principles.

If there are changes, a change management process that evaluates each change individually is NOT enough. Instead, a change should be related to other changes.

For each change that is implemented, either

• take another requirement out of scope, or
• increase the duration of the project.

During development, individual elements can therefore be replaced by elements that are equivalent in terms of development effort. Scrum supports this with the Product Backlog–by pulling a backlog item up, the other elements automatically have less urgency.

If you want to know more about agile contract design, I recommend you have a look at Money for Nothing and Your Change for Free.

To get the basics of agile software development right, visit my online course. If you want to keep up with what I'm doing or drop me a note, follow me on dev.to, LinkedIn or Twitter. Or visit my GitHub project.

Following my recent article on applying Reinforcement Learning to real life problems, I decided to demonstrate this with a small example. The aim is to create an algorithm that can find a suitable choice of food products to fit within a budget and meet my personal preferences.

I have also posted the description, data and code kernel to Kaggle and this can be found here.

Please let me know if you have any questions or suggestions.

Photo: Pixabay

Aim

When food shopping, there are many different products for the same ingredient to choose from in supermarkets. Some are less expensive, others are of higher quality. I would like to create a model that, for the required ingredients, can select the optimal products required to make a meal that is both:

1. Within my budget
2. Meets my personal preferences

To do this, I will first build a very simple model that can recommend the products that are below my budgets before introducing my preferences.

The reason we use a model is so that we could, in theory, scale the problem to consider more and more ingredients and products that would cause the problem to then be beyond the possibility of any mental calculations.

Method

To achieve this, I will be building a simple reinforcement learning model and I’ll use Monte Carlo learning to find the optimal combination of products.

First, let us formally define the parts of our model as a Markov Decision Process:

• We have a finite number of ingredients required to make any meal and are considered to be our States
• There are the finite possible products for each ingredient and are therefore the Actions of each state
• Our preferences become the Individual Rewards for selecting each product, we will cover this in more detail later

Monte Carlo learning takes the combined the quality of each step towards reaching an end goal and requires that, in order to assess the quality of any step, we must wait and see the outcome of the whole combination. This process is repeated over and over again in episodes with many different products until is finds the selection that appears to lead to a positive outcome repeatedly. This is the reinforcement learning process where our environment is simulated based on the knowledge about costs and preferences we obtained.

Monte Carlo is often avoided due to the time required to go through the whole process before being able to learn. However, in our problem it is required as our final check when establishing whether the combination of products selected is good or bad is to add up the real cost of those selected and check whether or not this is below or above our budget. Furthermore, at least at this stage, we will not be considering more than a few ingredients and so the time taken is not significant in this regard.

_[https://www.tractica.com/artificial-intelligence/reinforcement-learning-and-its-implications-for-enterprise-artificial-intelligence/](https://www.tractica.com/artificial-intelligence/reinforcement-learning-and-its-implications-for-enterprise-artificial-intelligence/" rel="noopener" target="blank" title=")

Sample Data

For this demonstration, I have created some sample data for a meal where we have 4 ingredients and 9 products, as shown in the diagram below.

We need to select one product for each ingredient in the meal.

This means we have 2 x 2 x 2 x 3 = 24 possible selections of products for the 4 ingredients.

I have also included the real cost for each product and V_0.

V_0 is simply the initial quality of each product to meet our requirements and we set this to 0 for each.

Diagram showing the possible product choices for each ingredient

First, we import the required packages and data.

Applying the Model in Theory

For now, I will not introduce any individual rewards for the products. Instead, I will simply focus on whether the combination of products selected is below our budget or not. This outcome is defined as the Terminal Reward of our problem.

For example, say we have a budget of £30, then the choice:

a1→b1→c1→d1

Then the real cost of this selection is:

£10+£8+£3+£8 = £29 < £30

And therefore, our terminal reward is:

R_T=+1

Whereas,

a2→b2→c2→d1

Then the real cost of this selection is:

£6+£11+£7+£8 = £32 > £30

And therefore, our terminal reward is:

R_T=−1

For now, we are simply telling our model whether the choice is good or bad and will observe what this does to the results.

Model Learning

So how does our model actually learn? In short, we get our model to try out lots of combinations of products and at the end of each tell it whether its choice was good or bad. Over time, it will recognise that some products generally lead to getting a good outcome while others do not.

What we end up creating are values for how good each product is, denoted V(a). We have already introduced the initial V(a) for each product, but how do we reach go from these initial values to actually being able to make a decision?

For this, we need an Update Rule. This tells the model, after each time it has presented its choice of products and we have told it whether it’s selection is good or bad, how to add this to our initial values.

Our update rule is as follows:

This may look unusual at first but in words we are simply updating the value of any action, V(a), by an amount that is either a little more if the outcome was good or a little less if the outcome was bad.

G is the Return and is simply to total reward obtained. Currently in our example, this is simply the terminal reward (+1 or -1 accordingly). We will reintroduce this later when we include individual product rewards.

Alpha, αα, is the Learning Rate and we will demonstrate how this effects the results more later but just for now, the simple explanation is: “The learning rate determines to what extent newly acquired information overrides old information. A factor of 0 makes the agent learn nothing, while a factor of 1 makes the agent consider only the most recent information.” (https://en.wikipedia.org/wiki/Q-learning)

Small Demo of Updating Values

So how do we actually use this with our model?

Let us start with a table that has each product and its initial V_0(a):

We now pick a random selection of products, each combination is known as an episode. We also set α=0.5α=0.5 for now just for simplicity in the calculations.

For example:

Therefore, all actions that lead to this positive outcome are updated as well to produced the following table with V1(a):

So let us try again by picking another random episode:

Therefore, we can add V2(a) to our table:

Action Selection

You may have noticed in the demo, I have simply randomly selected the products in each episode. We could do this, but using a completely random selection process may mean that some actions are not selected often enough to know whether they are good or bad.

Similarly, if we went another way and decided to select the products greedily, i.e. to ones that currently have the best value, we may miss one that is in fact better but was never given a chance. For example, if we chose the best actions from V2(a) we would get a2, b1, c1 and d2 or d3 which both provide a positive terminal reward therefore, if we used a purely greedy selection process, we would never consider any other products as these continue to provide a positive outcome.

Instead, we implement epsilon-greedy action selection where we randomly select products with probability ϵ, and greedily select products with probability 1−ϵ1−ϵ where:

This means that we are going reach the optimal choice of products quickly, as we continue to test whether the ‘good’ products are in fact optimal. But it also leaves room for us to also explore other products occasionally, just to make sure they aren’t as good as our current choice.

Building and Applying our Model

We are now ready to build a simple model as shown in the MCModelv1 function below.

Although this seems complex, I have done nothing more than apply the methods previously discussed in such a way that we can vary the inputs and still obtain results. Admittedly, this was my first attempt at doing this and so my coding may not be perfectly written but should be sufficient for our requirements.

To calculate the terminal reward, we currently use the following condition to check if the total cost is less or more than our budget:

The full code for the model is too large to fit here nicely, but can be found at the linked Kaggle page.

We now run our model with some sample variables:

In our function, we have 6 outputs from the model:

• Mdl[0]: Returns the Sum of all V(a) for each episode
• Mdl[1]: Returns to Sum of V(a) for the cheapest products, possible to define due to the simplicity of our sample data
• Mdl[2]: Returns the Sum of V(a) for the non-cheapest products
• Mdl[3]: Returns the optimal actions of the final episode
• Mdl[4]: Returns the data table with the final V(a) added for each product
• Mdl[5]: Shows the optimal action at each episode

There is a lot to take away from these, so let us go through each and establish what we can learn to improve our model.

Optimal actions of final episode

First, let’s see what the model suggests we should select. In this run it suggests actions, or products, that have a total cost below budget which is good.

However, there is still more that we can check to help us understand what is going on.

First, we can plot the total V for all actions, and we see that this is converging, which is ideal. We want our model to converge so that as we try more episodes we are ‘zoning-in’ on the optimal choice of products. The reason the output converges is because we are reducing the amount it learns each time by a factor of αα, in this case 0.5. We will show later what happens if we vary this or don’t apply this at all.

We have also plotted the sum of V for the products we know are cheapest, based on being able to assess the small sample size, and the others separately. Again, both are converging positively although the cheaper products appear to have slightly higher values.

So why is this happening and why did the model suggest the actions it did?

To understand that, we need to dissect the suggestions made by the model at each episode and how this relates to our return.

Below, we have taken the optimal action for each state. We can see that the suggested actions do vary greatly between episodes and the model appears to decide which is wants to suggest very quickly.

Therefore, I have plotted the total cost of the suggested actions at each episode and we can see the actions vary initially then smooth out and the resulting total cost is below our budget. This helps us understand what is going on greatly.

So far, all we have told the model is to provide a selection that is below budget and it has. It has simply found a answer that is below the budget as required.

So what is the next step? Before I introduce rewards I want to demonstrate what happens if I vary some of the parameters and what we can do if we decide to change what we want our model to suggest.

Effect of Changing Parameters and How to Change Model’s Aim

We have a few parameters that can be changed:

1. The Budget
2. Our learning rate, α
3. Out action selection parameter, ϵ

Varying Budget

First, let us observe what happens if we make our budget either impossibly low or high.

A small budget means we only obtain a negative reward means that we will force our V to converge negatively whereas a budget that is too high will cause our V to converge positively as all actions are continually positive.

The latter seems like what we had in our first run, a lot of the episodes lead to positive outcomes and so many combinations of products are possible and there is little distinction between the cheapest products from the rest.

If instead we consider a budget that is reasonably low given the prices of the products, we can see a trend where the cheapest products look to be converging positively and the more expensive products converging negatively. However, the smoothness of these is far from ideal, both appear to be oscillating greatly between each episode.

So what can we do the reduce the ‘spikiness’ of the outputs? This leads us onto our next parameter, alpha.

Varying Alpha

A good explanation of what is going on with our output due to alpha is described by stack overflow user VishalTheBeast:

“Learning rate tells the magnitude of step that is taken towards the solution.

It should not be too big a number as it may continuously oscillate around the minima and it should not be too small of a number else it will take a lot of time and iterations to reach the minima.

The reason why decay is advised in learning rate is because initially when we are at a totally random point in solution space we need to take big leaps towards the solution and later when we come close to it, we make small jumps and hence small improvements to finally reach the minima.

Analogy can be made as: in the game of golf when the ball is far away from the hole, the player hits it very hard to get as close as possible to the hole. Later when he reaches the flagged area, he choses a different stick to get accurate short shot.

So it’s not that he won’t be able to put the ball in the hole without choosing the short shot stick, he may send the ball ahead of the target two or three times. But it would be best if he plays optimally and uses the right amount of power to reach the hole. Same is for decayed learning rate.” — source

To better demonstrate the effect of varying our alpha, I will be using an animated plot created using Plot.ly.

I have written a more detailed guide on how to do this here.

In our first animation, we vary alpha between 1 and 0.1. This enables us to see that as we reduce alpha our output smooths somewhat but it still pretty rough.

However, even though the results are smoothing out, they are no longer converging in 100 episodes and, furthermore, they output seems to alternate between each alpha. This is due to a combination of small alphas requiring more episodes to learn and out action selection parameter epsilon being 0.5. Essentially, the output is still being decided by randomness half of the time and so out results are not converging within the 100 episode frame.

Running this through our animated plots produces something similar to the following:

Varying Epsilon

With the previous results in mind, we now fix alpha to be 0.05 and vary epsilon between 1 and 0 to show the effect of completely randomly selecting actions to selecting actions greedily.

The graphs below show three snapshots from varying epsilon, but the animated version can be viewed in the Kaggle kernel.

We see that having a high epsilon creates very sporadic results. Therefore we should select something reasonably small like 0.2. Although have epsilon equal to 0 looks good because of how smooth the curve is, as we mentioned earlier, this may lead us to a choice very quickly but may not be the best. We want some randomness so the model can explore other actions if needed.

Increasing the Number of Episodes

Lastly, we can increase the number of episodes. I refrained from doing this sooner because we were running 10 models in a loop to output our animated graphs and this would have caused the time taken to run the model to explode.

We noted that a low alpha would require more episodes to learn so we can run our model for 1000 episodes.

However, we still notice that the output is oscillating, but, as mentioned before, this is due to our aim being simply to recommend a combination that is below budget. What this shows is that the model can’t find the single best combination when there are many that fit below our budget.

Therefore, what happens if we change our aim slightly so that we can use the model to find the cheapest combination of products?

Changing our Model’s Aim to Find the Cheapest Combination of Products

This aim of this it to more clearly separate the cheapest products from the rest, and it nearly always provides us with the cheapest combination of products.

To do this, all we need do is adapt our model slightly to provide a terminal reward that is relative to how far below or above budget this combination in the episode is.

This can done by changing the calculation for return to:

We now see that the separation between the cheapest products and the others is emphasised.

This really demonstrates the flexibility of reinforcement learning and how easy it can be to adapt the model based on your aims.

Introducing Preferences

So far, we have not included any personal preferences towards products. If we wanted to include this, we can simply introduce rewards for each product whilst still having a terminal reward that encourages the model to be below budget.

This can done by changing the calculation for return to:

So why is our return calculation now like this?

Well firstly, we still want our combination to be below budget so we provide the positive and negative rewards for being above and below budget respectively.

Next, we want to account for the reward of each product. For our purposes, we define the rewards to be a value between 0 and 1. MC return is formally calculated using the following:

γ is the discount factor and this tells us how much we value later steps compared to earlier steps. In our case, all actions are equally as important to reaching the desired outcome of being below budget so we set γ=1.

However, to ensure that we reach the primary goal of being below budget, we take the average of the sum of the rewards for each action so that this will always be less than 1 or -1 respectively.

Again, the full model can be found in the Kaggle kernel but is too large to link here.

Introducing Preferences using Rewards

Say we decided we wanted product a1 and b2, we could add a reward to each. Let us see what happens if we do this in the output and graphs below. We have changed out budget slightly as a1 and b2 add up to £21 which means there is no way to select two more products that would put it below a budget of £23.

Applying a very high reward forces the model to pick a1 and b2 then work around to find products that will put it under our budget.

I have kept in the comparison between the cheapest products and the rest to show that the model now is not valuing the cheapest once more. Instead we get the output a1, b2, c1 and d3 which has a total cost of £25. This is both below our budget and includes our preferred products.

Let’s try one more reward signal. This time, I give some reward to each but want it to provide the best combination from my rewards that still keeps us below budget.

We have the following rewards:

Running this model a few times shows that it would:

• Often select a1 as this has a much higher reward
• Would always pick c1, as the rewards are the same but it is cheaper
• Had a hard time selecting between b1 and b2 as the rewards are 0.5 and 0.6 but the costs are £8 and £11 respectively
• Would typically select d3 as being significantly cheaper than d1 even though reward is slightly less

Conclusion

We have managed to build a Monte Carlo Reinforcement Learning model to:

1. recommend products below a budget,
2. recommend the cheapest products, and
3. recommend the best products based on a preference that is still below a budget.

Along the way, we have demonstrated the effect of changing parameters in reinforcement learning and how understanding these enables us to reach a desired result.

There is much more that we could do, in my mind, the end goal would be to apply to a real recipe and products from a supermarket where the increased number of ingredients and products need to be accounted for.

I created this sample data and problem to better my understanding of Reinforcement Learning and hope that you find it useful.

Thanks for reading!

Sterling Osborne