TL;DR Check this back end and front end solutions for delegated transactions. It is universal for any token which supports the delegation of its functions. Read more below.

This mostly technical article provides a universal framework and a working solution for Ethereum tokens and applications that eliminates the need to pay fees in Ether, a problem that is practically killing the user experience of many blockchain applications.

Imagine spending dollars and then being asked to also hand over some Hryvnias as a transaction fee. That's how Ethereum tokens work so far.

In other words, for example, to transfer any Ethereum token (like Tether, DAI, BAT, DREAM, etc.), the user has to also spend some Ether (internal Ethereum platform currency). This introduces a big inconvenience that prevents the mass adoption of DApps: users have to purchase multiple currencies instead of just one to interact with the blockchain network.

The Problem

Tokens, as we imagine them today are just fuel for applications and services on top of blockchain networks. Organizations create their own tokens (using ICOs, IEOs, etc) and run services/applications that utilize them, introducing their own micro-economy (widely known as a token economy). But almost every token turns out to be quite a complex currency itself. By design of how blockchain networks work, in order to do something with your tokens, you also need another currency — often Ether (for Ethereum) to be able to transfer tokens.

To illustrate the problem, let's look into how users come to use different blockchain-powered services and applications like:

• Trickle - where you create secure, hourly-based contracts with an untrusted party in any token
• Loom - where you use Loom tokens to create sidechains in Loom Network
• Cryptokitties.co - where you breed, trade and transfer kitties (ERC721 tokens)
• Others (there are a lot!)

All these applications use tokens, as well as they require you to purchase Ether. The complexity of using crypto tokens as we know them today is one of the biggest reasons why 99% of crypto startups fail (or avoid adopting real crypto, for example, by replacing it with virtual coins).

As you may already know, the harder it is to use the application, the fewer users it will get right from the beginning. This is something known as The User Onboarding Funnel, which is still a big pain for blockchain-powered applications and services:

The typical user onboarding funnel of a decentralized, blockchain-based application

To understand why I put 0.001% of users prior to the service use, let's see what exactly purchasing some Ether means:

• Creating a crypto wallet
• Registering on Exchange (and learning all the exchange rules, including country policies!)
• Passing KYC (though it's getting easier, still, many countries have limited access to exchanges)
• Purchasing a minimum allowed amount of Ether (usually, it's whopping $50 while you need just nearly $0.05 to perform one or two transactions)
• Withdrawing Ether to your wallet
• Not to mention reading lengthy guides on how to perform all these steps properly

Instead of just:

• Creating a crypto wallet

Of course, it highly depends on how the application or service is made. But, so far, there was no better simplification of the onboarding flow as just cutting crypto tokens from there, or making them fake, "virtual" currency with deposit and withdrawal function. Unfortunately, the latter approach is now the common one across all startups and companies adopting crypto, for many good reasons. Another reason could be a monetization strategy, but this is another big story worth a dedicated article (Interested? Comment out!).

Getting back to the transaction fees problem, we can state the following, which is hard to argue with.

It is natural for the user to purchase only the cryptocurrency they really need (for instance, tokens: Tether, DAI, BAT, DREAM, etc.), and they would normally expect to pay any transaction fees in this cryptocurrency.

So why not just allow them to do so? Because it's quite complex indeed. Let's see why, and how this has just got easier with our open-sourced solution (at least for Ethereum).

Existing Approaches

From the very beginning of blockchain existence, there were a couple of solutions that could simplify the user onboarding flow to the flow depicted below, avoiding the step of purchasing an intermediate currency like Ether. Still, creating a blockchain wallet is not an easy step, but some users who do understand the value of the application/service go through this step quite well.

The user onboarding funnel with delegated transactions

The solution which allows to avoid using intermediate currencies (Ether for Ethereum) is called "delegated transactions", or "meta transactions".

In short, delegated transaction, or "meta transaction" in blockchain is the type of transaction which performs an intended action on one account's behalf, while it is conducted (published) by another account (delegate), who actually pays fees for the transaction.

There are multiple approaches around the internet of the generalized concept of delegated transactions I am presenting in this article. But it seems that none of them are still widely adopted, as these approaches are quite complex by its nature, very specific as for the implementation, as well as some of them are quite complex to standardize. To be more constructive, existing approaches can be divided into 3-4 groups: those which use proxy smart contracts, those which embed delegation into a smart contract itself and, theoretically, there is an opportunity for the blockchain-native implementation (say, Ethereum 2.0).

1. Delegated transactions approaches which use proxy-contracts

Proxy contracts, or, in this context, identity contracts are tiny contracts deployed to replace the user account which wants to avoid paying fees. This smart contract is programmed to act as a wallet, as well as a "caller" (sender) of other smart contract's functions. The key is that it is a delegate account that triggers all the actions, while the true "owner" of this smart contract is another user. The user just generates correct signatures in order to control their funds stored on a smart contract address (= in their wallet).

A visualization of how identity contracts look like

Pros of this approach:

• It works with any tokens and contracts which are already deployed to the network

Cons of this approach:

• Users don't see tokens in their wallet, because they are physically on an identity smart contract
• As a result, a need to develop custom UIs and custom tools/wallets
• Identity smart contract deployment and assignment initial fees, as opposed to no fees at all
• Requires a comprehensive standard to be widely adopted

2. Semi-delegated transactions via "Operator" pattern (ERC777)

There is a token standard that describes this approach — ERC777. In short, any token holder can authorize any other account to freely manage their tokens. I won't call it delegated transactions but nevertheless, I need to mention that, as here we somewhat delegate control over your tokens to other accounts.

A visualization of ERC777 standard's "operator" pattern

Pros of this approach:

• Standardized

Cons of this approach:

• Highly centralized around the "operator" accounts
• Weak security due to operators have 100% control over your tokens
• Initial fees for approval transaction
• Requires additional UIs/tools development

3. Delegated transactions embedded directly into a (token) smart contract

Just the same as it is possible to implement custom fees in a proxy smart contract, paying fees in tokens can be implemented directly in a token smart contract. For example, using the approach I described in my previous article, it is possible to implement a function in a smart contract, which will transfer tokens accepting the user's signature, instead of requiring the user to call this function directly. We have implemented this approach in our DREAM Token, which is used on our dreamteam.gg platform.

A visualization of how embedding delegation into the token contract looks like

As you may notice, in contrast to the previous approach there is no identity contract anymore, and there is an optional way to call other smart contracts directly from the token contract.

Pros of this approach:

• Users see their tokens as usual on their wallet's balance
• No initial fees for account initialization
• May not even require a standard (continue reading)

Cons of this approach:

• If you have a (token) smart contract that is already deployed to the network, you cannot apply this approach to it directly. While you can always deploy a new token and, for example, a "migration" utility, which will allow other users to swap tokens (burn the old token and mint a new one)
• Because a standard for this approach is yet not well-defined, implementation can drastically vary
• A need to develop custom UI/tools for delegated transactions (continue reading — solved!)

4. Delegated transactions on the (blockchain) platform level

This is far the best one of all the described approaches above but also the one which is not implemented anywhere yet (by anywhere I mean the most popular blockchain platforms). There is a hope that its support comes with Ethereum 2.0 release, or at least I've heard from Vitalik that they are in progress with something cool there.

Theoretically, we can imagine this approach as being able to make an "offline" signature of two transactions at a time: one which does something useful for the signing account which wants to avoid paying fees (for example, transferring tokens) and another one which does something useful for the delegate (for example, paying fee in tokens to the account which executes these two transactions).

A visualization of how platform-native delegated transactions could look like

But the problem is, regarding Ethereum 2.0, this feature has a chance to land only in 2022 or even later. I also suppose that this feature will still require a dedicated back end (similar to the one which is introduced within this article), as it is hard to imagine how miners will accept fees in tokens. Simply put, if some of them refuse to accept fees in tokens than it makes little sense to do it on a "mining" level at all, not to mention how much would it take to track all token prices and volumes across exchanges, in a decentralized manner.

Pros of this approach:

• No need to change smart contracts that were already deployed
• No initial fees for account initialization
• May not even require a custom UI/tools if standardized

Cons of this approach:

• Most likely, will still require a centralized back end (the "delegate")
• Not yet implemented on a platform level (as of 2019)

The Solution

From the four approaches above, except for the platform-level approach which is yet to be implemented and standardized in 2022+, the most appealing one is the third approach, where we embed delegated functions directly to the token smart contract. Thus, we save the standard token paradigm allowing wallets to normally work with the smart contract and have no need to wait until delegated transactions will land natively in one of the top blockchain platforms. We will stick to this approach and make it universal just below.

Delegated transactions support programmed right in the token smart contract is awesome. But how to deal with its cons? In fact, the only problem which is tough to deal with (as you cannot modify existing smart contracts), you will need to deploy a new token smart contract if you have already deployed it without delegated functions (for instance, standard ERC20 or ERC721 tokens). The next step, in this case, would be adding a way to migrate tokens from one smart contract to another. For example, you can decide to implement one more function in the new smart contract that will allow token holders to migrate their assets from the old smart contract.

Token migration function implementation can vary, starting from implementing receiveApproval in the new token, if the previous token supports approveAndCall, or ending with utilizing approve + transferFrom framework if you have just a bare minimal ERC20 (the user _approve_s tokens to the new token contract address and then calls a function in the new contract which burns old tokens and mints new ones — but this requires a standard fee for the user for the approval transaction). Actually, there is more: you can decide not to burn old tokens but to "lock" them on a new token smart contract, minting new tokens — this opens an opportunity to implement two-sided token migration, which is awesome — you won't need to list the "new" token on the exchange, while the users will still be able to send the old token to exchanges without fees in Ether! If you are interested, please fill the issue here if you want to know more details on how to do it, because this approach is worth a whole new article.

In my previous article, I provided an example of the token smart contract which supports delegation of such functions like transfer, transferFrom, approve and approveAndCall. Exactly these "delegated" functions allow users to pay fees in tokens, instead of Ether.

How delegation works in Ethereum, in short. Read more in this article.

But that wasn't enough to start the mass adoption. In this article, I am providing a complete universal back end solution (Transaction Publisher in the picture above), as well as a configurable widget (check it here), which allows you to replace Ether fees for token fees today.

Some key points before we dive in:

• This delegated transactions back end is made to be universal, or standard-free, meaning that you can have any implementation of delegated functions and use any signature standard(s) in your token. From the back end standpoint, you just need to write a manifest file for your token, describing its usage.
• Currently, converting collected fees in tokens back to Ether is a manual action on exchanges. But it could be a potential improvement for automation in the future (if needed).

The Concept Behind the Universal Solution

What does it mean that the token supports delegated transactions? Let's look at it using the ERC20 standard token as an example.

Smart Contract

As for the token smart contract, the approach is quite straightforward. In addition to every method like transfer(to, value) which we want to be "delegatable", we add a companion function which, instead of inspecting msg.sender, accepts the signature of a user and does the same what the original function meant to do by validating this signature inside the smart contract. Thus, for example, for transfer(to, value) function we can add transferViaSignature(to, value, ...aditionalParams) function. As you know from public-key cryptography, no one can create a valid signature except private key owner, so that's why this approach is as secure as Ethereum itself.

And the coolest part is that the delegated function implementation, as well as its signature doesn't matter much, from the delegate back end standpoint. You can even decide to implement one "call by signature" function for all other functions that the smart contract supports. Delegate back end just need to know how to call this function, which is solved by providing an off-chain contract manifest for the delegate back end. For example, the argument additionalParams in transferViaSignature can vary and can include anything from this list, if not more: fee, fee recipient address, expiration timestamp, a signature standard used, a signature itself, nonce number or any other unique delegated transaction ID and so on. Regarding the smart contract design, in order to understand why exactly these arguments, read my previous article.

We also want to allow "delegates" to earn something in order to cover their Ether spending, as well as to be profitable. Thus, we have to add a fee, but a much more natural fee than Ether: a fee in the token itself. So that, for example, if you need to transfer 100 tokens, you pay 3 more tokens to the delegate depending on its price and network conditions to perform a transfer, and this should be preserved in a smart contract logic.

Back End

All right, now we have a token that allows transferring someone else's tokens by using their signature. Now, the crucial part is to automate the process of requesting and publishing such transactions. And here where our open-sourced back end (and a front end) kicks in.

Below is the sequence diagram describing how front end (client) communicates with the back end from the delegated transaction request to its publishing to the network:

1. (hidden on the diagram) The client requests information from the delegated back end to understand which contracts does it support, as well as which functions can it delegate.
2. The client requests a particular smart contract's function to be delegated. Most importantly, the back end returns the fees it charges and a data to be signed by the client.
3. The client signs the data in their wallet. Signing is a free operation, unlike publishing transaction to the network.
4. The client sends their signature back, thus confirming their intent to perform this particular delegated transaction. The back end validates this transaction against the current network.
5. Finally, the back end publishes a transaction to the network.
6. (hidden on the diagram) The client constantly polls the back end for the delegated request status until it receives a mined status. Note: it is important to poll the back end instead of using a transaction hash to understand when the transaction is mined. It is a very common case when the gas price suddenly increases, and, in order for the transaction to be mined quickly, the back end may republish it with a higher gas price. Though it is currently not implemented, it is very likely to be implemented soon.

Sequence diagram representing the simplified flow of how delegated transaction is delivered to the network

This approach is universal, and only requires the manifest file for the back end to understand how to calculate fees and which signature standard to use on the client side. Here is another visualization of the components of the system and their interaction sequence:

Component diagram

We've provided a comprehensive documentation for this solution. You can check how the back end API is structured, as well as find the token manifest file which describes how to work with a particular token contract. We encourage you to contribute your own tokens there!

And you don't need much setup: it's already there with the universal front end!

Front End

Open-sourced front end part of the delegated transactions is the user interface which is set up for every token: just run your delegated transactions back end and you are ready to go!

What it looks like

It is made to be an embeddable widget, which will guide the user through the procedure of sending tokens. You can plug any back end, token or call any token function with it by utilizing additional URL parameters you can specify.

Using this widget, and by implementing something similar to widely used, but not standardized approveAndCall function in your token smart contract, you will be able to call other smart contracts with arbitrary data by paying fees in tokens!

Here is a quick guide for you if you want to play with this UI yourself:

1. Access the widget via this link.
2. It will ask you to switch to the Kovan test network.
3. Get some test Ether using any available Kovan faucet.
4. Use test Ether to mint some test tokens: call mintTokens function in a token smart contract which will give you 10 test tokens.
5. Now, get back to the widget and try to transfer these tokens!

If you open up the browser's developer tools, you may notice that there are a couple of back ends connected by default — they provide the front end with all required information to make a delegated request according to given widget URL parameters. All backends are requested during the widget load and, if any of them can provide a delegation for a particular contract's function, then the widget requests additional information: fees, supported signatures, etc. If there are multiple back ends which can delegate the same contract function, all of them are requested and the back end which provides the best fee will be used for the transaction.

Transaction mining time is seemingly fixed, but it can vary because of the network conditions. The back end uses an actual network fee when calculating the token fee, however, it may change before the user decides to execute the transaction. Thus, the "underpriced" transaction is submitted to the network and can be pending for a while. While the back end is currently not programmed to deal with this case, it might be implemented in future — transactions will be republished with higher gas fees in case of the network fee increases. But, we will also need to count this into the token fee.

Signature Standards

The last question which you may be wondering is — which signature standard to use for your token. There are several available: _ethsign (deprecated), _ethpersonalSign (note that old Trezor and Ledger produce a different signatures because of ambiguity in a standard, so you may want to include both), _ethsignTypedData (deprecated), _eth_signTypedDatav3 and so on. I would recommend supporting at least two: ageless _ethpersonalSign and new _eth_signTypedDatav3 (as of 2019).

Signature standards comparison — what the user sees

The front end is programmed to always prefer the user-readable standard like eth_signTypedData_v3 to any others eth_personalSign. So if your token supports many signature standards, and you added all of them to the manifest file of your token, it will display eth_signTypedData_v3 prompt first.

Conclusion

Delegated transactions are great: they solve one of the biggest problems of blockchain application adoption, which eases the mass adoption of crypto overall. I will put a couple of thesis in a Q&A format here for you to answer the last questions that you may still have after reading this article:

• Our open-source solution is free to use and production-ready, feel free to apply it to your applications or tokens!
• The described approach does not compromise security nor centralization. Think this way: the centralized back end is only a helper for someone who wants to transfer tokens without fee in Ether. If the back end is hacked, or it is just unavailable, there's no problem to interact with the network just as it was before, by paying fees in Ether. As well as the back end cannot harm or trick the user to steal their tokens when a proper signature standard is used (it's up to your token implementation).
• There is a way to support delegated transactions for existing, already-deployed tokens. However, it requires the additional Ether-consuming step to migrate existing tokens to a new token contract. And, by programming a new token contract properly, as well as designing your application to work with both tokens you can even avoid a need to list a new token on exchanges.
• By using the existing tokens as an example, which is available in delegated transactions back end and front end repositories, you can produce your own manifest for your own token.
• Read the instructions on how to set up your own back end for a token, and then add it to the URL of your widget (or commit to the open-source repository).
• Have a token which already supports delegated transactions? Plug it into our UI with these three quite simple steps: (1) create a manifest for your token and put your token abi file while setting up the delegate back end, (2) run this back end, exposing a public API URL and (3) use URL parameters in a widget to reference your back end or commit it directly to our open-source repository. Read more about it in GitHub's readme file.

I hope that was a really helpful piece of information for all the searchers of incredible. Feel free to contact me or fill the issue here if I missed something. Have fun, let the token economy be simple!

Article Highlights:

• How to store private keys on the back end without security concerns
• How to properly design smart contracts and what to “decentralize”
• Decentralized and semi-centralized applications architecture examples
• How to deal with low-level stuff like network load and failures

It’s going to be big, let’s do it!

Decentralized Programs and Blockchain

Despite the fact that blockchain is facing a lot of adoption and regulation difficulties today, it’s a kind of technology which is here to stay, whether it’s blockchain, hashgraph, tempo or any other distributed ledger technology still to come, regardless of the algorithm.

The main value that blockchain and other similar technologies bring can be generalized as follows: they allow people to write and run programs which, practically, cannot be changed after creation nor tampered with during execution. In other words, these programs always run as designed, and no single party can influence their behavior.

This definition is valid for many cryptocurrencies that exist today if we consider them as programs that define how coins can be transferred back and forth. This also explains why cryptocurrencies and many kinds of tokens have real value: they cannot be generated out of thin air, by their defined “underlying programs”.

Ethereum/EOS/Tron/… platforms, in contrast to Bitcoin, implement a more complex program layer, which in turn implements the execution environment allowing anyone to write their own decentralized programs on top of the platform. This user-defined programs always run as designed, without any exceptions, and their security is guaranteed by the platform.

Decentralized Applications

These secure and unchangeable programs running on a decentralized network in combination with traditional front-end and back-end technologies are what we call decentralized applications (ÐApps) today. Through some of them can be semi-centralized, a major part of activities in the truly decentralized application should happen out of a central party’s control.

If someone asked me to draw how DApps work today, I would probably have drawn this

To imagine what we call decentralized applications today, take any existing centralized web resource like YouTube or Instagram as an example and imagine that instead of a password-protected centralized account you have your “crypto identity” bound to the web/mobile resource.

That’s what Wallet Software provides you. The private key from this identity (a secret, having which, you can act on behalf of this identity) is stored on your local device and never goes online, making no one able to control this identity but you. With this identity, you can perform different actions in both centralized (web resource controlled by a central authority) and decentralized (which is a different network from the traditional www, the goal of which is to eliminate the central authority) networks, using the website as an access point and/or as a graphical user interface. The whole point of this “crypto identity” is that your actions are cryptographically secured, and no one is able to change what have you signed nor your signature.

Today, the computational and storage capabilities of fault-tolerant decentralized networks like Ethereum, EOS or Tron are limited. If they were scalable, we could use decentralized networks to store the whole decentralized application, including its graphical user interface, data and business logic. In this case, we would call these applications truly decentralized/distributed.

However, because those networks are not scalable today, we combine different approaches to achieve the maximum decentralization level for our applications. The “traditional” back end as we know it isn’t going anywhere. For instance:

• We use back end to host front end for a decentralized application.
• We use back end for integrations with any other existing technologies and services. Real, world-class applications cannot live in an isolated environment.
• We use back end to store and process anything big enough for a decentralized network (blockchain in particular). Practically, the whole application and its business logic are stored somewhere in the world, excluding the blockchain part only. Not to mention, IPFS and similar storage layers cannot guarantee the accessibility of files, hence we cannot rely on them without hosting the files ourselves either. In other words, there’s always a need for a dedicated running server.

There’s no way of building a secure and partially decentralized application without using a solid back end as of today, and the whole point of this article is to explain how to do this right.

(De)centralization and Tokens

It so happens that almost all decentralized applications today are built around so-called tokens — custom-built (or just simply cloned) cryptocurrencies that drive a particular decentralized application. Tokens are simply a programmable currency or assets, that’s it.

While token smart contracts determine how users can transfer tokens, application smart contracts can extend everything missing in token smart contracts. Both smart contracts run on top of decentralized networks

Usually, a token is a “smart contract” (a custom program) written on top of the decentralized platform like Ethereum. By owning some tokens you are basically able to get different services on a web resource or mobile app, and trade this token for something else. The key point here is that the token lives on its own and it is not controlled by a central authority.

There are many examples of applications that are build around tokens: from numerous collectible games like CryptoKitties (ERC721 tokens) to more service-oriented apps like LOOM Network, or even browsers like Brave and gaming platforms like DreamTeam (ERC20-compatible tokens).

Developers themselves determine and decide how much control they will (or won’t) have over their applications. They can build the whole application’s business logic on top of smart contracts (like CryptoKitties did), or, they can make no use of smart contracts at all, centralizing everything on their servers. However, the best approach is somewhere in the middle.

Back End for Decentralized Networks

From a technical point of view, there has to be a bridge that connects tokens and other smart contracts with the web/mobile application.

In today’s fully decentralized applications, where clients interact with smart contracts directly, this bridge is narrowed down to a JSON RPC API capabilities of public APIs or node pools like Infura, which in turn are forced to exist because of the fact that not every device can run and support its individual network node. However, this API provides an only basic and very narrow set of functions, which barely allow making simple queries nor efficiently aggregate data. Because of this, eventually, custom back end kicks in, making the application semi-centralized.

The whole interaction with the decentralized network can be narrowed down to just one or two points, depending on the application needs:

1. Listening to the network events (like token transfers) / reading the network state.
2. Publishing transactions (invoking state-changing smart contract functions like token transfer).

Implementing both of these points is quite tricky, especially if we want to build a secure and reliable back-end solution. Here are the main points which we are going to break down below:

• First of all, in Ethereum, events retrieval is not production-ready out of the box. Because of multiple reasons: network nodes can fail while fetching a large number of events, events can disappear or change because of network forks, etc. We have to build an abstraction layer to sync events from the network and guarantee their reliable delivery.
• The same for transaction publishing, we have to abstract Ethereum’s low-level stuff like nonce counters and gas estimations, as well as transaction republishing, providing a reliable and stable interface. Moreover, transaction publishing implies using private keys, which requires advanced back-end security.
• Security. We are going to take it seriously and face that it’s impossible to guarantee that private keys won’t ever be compromised on a back end. Luckily, there is an approach to designing a decentralized application without even a need for back-end accounts to be highly secured.

In our practice, all of this made us create a robust back-end solution for Ethereum which we name Ethereum Gateway. It abstracts other microservices from Ethereum’s fun and provides a reliable API to work with it.

As a fast-growing startup, we cannot disclose the complete solution (just yet) for obvious reasons, but I am going to share everything that you need to know to make your decentralized application work flawlessly. However, if you have any specific questions or inquiries, feel free to contact me. Comments to this article are much appreciated as well!

_Back End Monitoring for Ethereum. The monitor demonstrates activities mainly regarding our [recurring billing feature](https://github.com/dreamteam-gg/smart-contracts/blob/master/contracts/token/TokenRecurringBilling.md" rel="noopener" target="blank" title=") (though you can see peaks happening each hour).

Decentralized Applications Architecture

This part of the article highly depends on the needs of a particular decentralized application, but we will try to sort out some basic interaction patterns on top of which these applications are built (ÐPlatform = Decentralized Platform = Ethereum/EOS/Tron/Whatever):

Client ⬌ ÐPlatform: fully decentralized applications.

The client (browser or mobile application) talks to the decentralized platform directly with the help of Ethereum “wallet” software like Metamask, Trust or hardware wallets like Trezor or Ledger. Examples of DApps build in such manner are CryptoKitties, Loom’s Delegated Call, crypto wallets themselves (Metamask, Trust, Tron Wallet, others), decentralized crypto exchanges like Etherdelta and so on.

ÐPlatform ⬌ Client ⬌ Back End ⬌ ÐPlatform: centralized or semi-centralized applications.

The client interaction with the decentralized platform and the server has little in common. The good example of this is any (centralized) crypto exchange today, like BitFinex or Poloniex: the currencies you trade on exchanges are simply recorded in the traditional database. You can “top up” your database balance by sending assets to a specific address (ÐPlatform ⬌ Client) and then withdraw assets after some actions in the app (Back End ⬌ ÐPlatform), however, everything you do in terms of a “ÐApp” itself (Client ⬌ Back End) does not imply your direct interaction with the ÐPlatform.

Another example is Etherscan.io, which uses semi-centralized approach: you can do all useful decentralized actions there, but the application itself just doesn’t make sense without their comprehensive back end (Etherscan continuously syncs transactions, parses data and stores it, ultimately providing a comprehensive API/UI).

Something in between: still, centralized or semi-centralized applications.

The above approaches combined. For example, we can have an application which provides various services in exchange for crypto, allowing you to log in and sign information with your crypto identity.

Hopefully, the interaction pattern of fully decentralized applications (Client ⬌ ÐPlatform) does not raise any questions. By relying on such amazing services like Infura or Trongrid one can simply build an application which doesn’t require a server at all. Almost all client-side libraries like Ethers.js for Ethereum or Tron-Web for Tron can connect to these public services and communicate with the network. However, for more complex queries and tasks, you may need to allocate your own server anyway.

The rest of the interaction patterns which involve back end make things more interesting and complex. To put all these in a picture, let’s imagine a case where our back end reacts to some event in the network. For example, the user publishes an allowance transaction which gives us permission to charge them. To make a charge, we have to publish the charge transaction in response to the emitted allowance event:

An example flow of server’s reaction to the user’s action in the decentralized network

From the back end point of view here’s what happens:

1. We listen to a particular network event by continuously polling the network.
2. Once we get an event, we perform some business logic and then decide to publish a transaction in response.
3. Prior to publishing the transaction, we want to ensure that it will likely be mined (in Ethereum, the successful transaction gas estimation means there are no errors against the current network state). However, we can’t guarantee that the transaction will be mined successfully.
4. Using a private key, we sign and publish the transaction. In Ethereum we also have to determine the gas price and gas limit of the transaction.
5. After publishing the transaction, we continuously poll the network for its status.
6. If it takes too long and we can’t get the status of the transaction, we have to re-publish it or trigger a “fail scenario”. Transactions can be lost for various reasons: network congestion, dropping peers, network load increase, etc. In Ethereum, you can also consider re-signing a transaction with a different (actual) gas price.
7. After we finally get our transaction mined, we can perform more business logic if needed. For example, we can notify other back end services about the fact of the transaction being completed. Also, consider waiting for a couple of confirmations prior to making final decisions regarding the transaction: the network is distributed and hence the result can change in a matter of seconds.

As you can see, there’s a lot going on! However, your application may not require some of these steps, depending on what you are trying to achieve. But, building a robust and stable back end requires having a solution for all the problems mentioned above. Let’s break this down.

Decentralized Applications Back End

Here I want to highlight some of the points which arise most of the questions, namely:

• Listening to network events and reading data from the network
• Publishing transactions & how to do it securely

Listening to Network Events

In Ethereum, as well as in other decentralized networks, a concept of smart contract events (or event logs, or just logs) allows off-chain applications to be aware of what is happening in the blockchain. These events can be created by smart contract developers at any point of the smart contract code.

For example, within the well-known ERC20 token standard each token transfer has to log the Transfer event, thus letting off-chain applications know that there is a token transfer happened. By “listening” to these events we can perform any (re)actions. For instance, some mobile crypto-wallets send you a push/email notification when tokens are transferred to your address.

In fact, there’s no reliable solution for listening to network events out of the box. Different libraries allow you to track/listen to events, however, there are many cases when something can go wrong, resulting in lost or unprocessed events. To avoid losing events, we have to build a custom back end, which will maintain the events sync process.

Depending on your needs, the implementation can vary. But to put you in a picture here is one of the options of how can you build reliable Ethereum events delivery in terms of microservice architecture:

Reliable delivery of Ethereum events to all back end services

These components work in the following way:

1. Events sync back end service constantly polls the network, trying to retrieve new events. Once there are some new events available, it sends these events to the message bus. Upon successful event submission to the message bus, as for blockchain, we can save the last event’s block in order to request new events from this block next time. Keep in mind that retrieving too many events at once may result in always failing requests, so you have to limit the number of events/blocks you request from the network.
2. The message bus (for example, Rabbit MQ) routes the event to every queue which was set up individually for each back end service. Prior to event publishing, events sync back end service specifies the routing key (for example, a smart contract address + event topic), while consumers (other back end services) create queues which are subscribed for particular events only.

As a result, each back end service gets only those events it needs. Moreover, the message bus guarantees the delivery of all events once they are published to it.

Of course, you can use something else instead of the message bus: HTTP callbacks, sockets, etc. In this case, you’ll need to figure out how to guarantee callbacks delivery yourself: manage exponential/custom callback retries, implement custom monitoring and so on.

Publishing Transactions

There are a couple of steps we have to perform in order to publish a transaction to the decentralized network:

1. Preparing the transaction. Along with transaction data, this step implies requesting the network state in order to find out whether this transaction is valid and is going to be mined (gas estimation in Ethereum) and the transaction’s sequential number (nonce in Ethereum). Some of the libraries try to do this under the hood, however, these steps are important.
2. Signing the transaction. This step implies the usage of the private key. Most likely, here you’ll want to embed the custom private key assembly solution (for instance).
3. Publishing and republishing the transaction. One of the key points here is that your published transaction always has a chance to get lost or dropped from the decentralized network. For example, in Ethereum, the published transaction can be dropped if the network’s gas price suddenly increases. In this case, you have to republish the transaction. Moreover, you may want to republish the transaction with other parameters (at least with higher gas price) in order to get it mined as soon as possible. Thus, republishing the transaction can imply re-signing it, if the replacement transaction wasn’t pre-signed before (with different parameters).

The above points regarding Ethereum transaction publishing visualized

By utilizing the above approaches you can end up building something similar to the thing which is presented in the sequence diagram below. On this particular sequence diagram, I demonstrate (in general!) how the blockchain recurring billing works (there’s more in a linked article):

1. The user executes a function in a smart contract, which ultimately allows the back end to perform a successful charge transaction.
2. A back end service responsible for a particular task listens to the event of charge allowance and publishes a charge transaction.
3. Once the charge transaction is mined, this back end service responsible for a particular task receives an event from the Ethereum network and performs some logic (including setting the next charge date).

_The general sequence diagram of how [blockchain recurring billing](https://hackernoon.com/payments-of-tomorrow-decentralized-recurring-billing-47d126d895fd" rel="noopener" target="blank" title=") works, demonstrating the interaction between back-end services and Ethereum network

Back End Security & Smart Contracts

Transaction publishing always involves using a private key. You may be wondering if it is possible to keep private keys secure. Well, yes and no. There are numerous complex strategies and different types of software which allow storing private keys on the back end quite securely. Some private key storage solutions use geo-distributed databases, while others even suggest the use of special hardware. However, in any case, the most vulnerable point of a semi-centralized application is where the private key is assembled and used to sign a transaction (or, in case of special hardware, a point of triggering a transaction signing procedure). Hence, theoretically, there’s no 100% reliable solution which will enable bullet-proof protection from compromising stored private keys.

Now think this way. In many cases, you don’t even need to secure private keys on the back end that often. Instead, you can design smart contracts and the whole application in such a manner that a private key leak won’t affect their usual behavior. With this approach, it doesn’t matter how authorized accounts interact with the smart contract. They’re just “triggering” a smart contract to do its usual job, and the smart contract itself performs any required validation. I call it the “operational accounts pattern”.

Operational accounts pattern for decentralized applications, where you don’t need military-grade security for your back-end accounts

This way, in case of emergency:

• The only thing the attacker can steal is a tiny amount of Ether (as of Ethereum) deposited to the operational accounts for transaction publishing
• The attacker won’t be able to harm the smart contract logic nor anyone who is involved in the process
• Compromised operational accounts can be quickly replaced with other ones, however, this requires either the manual replacement (generating new accounts, and reauthorizing accounts in all smart contracts) or developing an additional solution which will do all the magic with a single transaction from a super-secure (hardware or multi-sig) master account.

For instance, in our recurring billing for Ethereum solution, no matter what happens on a back end, the recurring billing smart contract is designed in such a manner that we have a whole month of time for replacing the compromised accounts if any of them are compromised.

But still, if you want to get your back end private key storage as secure as possible, you can try using Vault with a great plugin for Ethereum which stores and manages Ethereum accounts (also, keep an eye on our open-source modules — we are about to release something similar soon). I am not going to dive deep into the details here, though you can visit the linked projects and learn from there yourself.

This isn’t even all I have to say. However, this article turned out to be much longer than I expected so I have to stop. Subscribe to my Medium / other networks if you’re interested in software, crypto, travel tips or just want to follow something interesting. Hope I’ve provided a big valuable piece of information and you’ll find it useful. Feel free to comment and spread this article!

Hack #1: Multiple email addresses within a single account

Have you ever wanted to register multiple times on a website, but still keep the same inbox for all the accounts? This simple trick works with about 95% of websites and services across the world. Even on Twitter and Instagram.

Some email providers, like gmail.com or live.com, allow you to put an extra dot between the letters of the email address. This changes nothing but the look of the address.

For example, say you own letgrandmaseat@gmail.com. You also own let.grandma.seat@gmail.com, let.grandmas.eat@gmail.com, l.e.t.g.r.a.n.d.m.a.s.e.a.t@gmail.com, and so on. All the emails sent to these addresses will appear in the same inbox. You can send emails back using any of these addresses as well — it just needs a bit of configuration.

This allows you to change the look of an email address, and use any of the iterations as a “new,” valid email. For example:

• You can register on websites multiple times using the same email account
• You can share different forms of your email address with different people. This way, when receiving emails from unknowns, you can find out how senders learned about your email address.

In addition, some email providers allow you to add a plus sign to your address followed by any arbitrary string:

• let.grandmas.eat+your.soup@gmail.com
• let.grandma.seat+a.baby@gmail.com

This increases the number of possible email address variations almost to infinity.

This can be extremely useful for grouping your incoming emails. You can set up a rule to group emails differently according to the address they come from. For example, you can register on some minor websites using the letgrandmaseat+trash@gmail.com address. Then, you can set up an inbox rule to move all those incoming emails to the “Low Priority” box.

Many services offering a trial version of their products accept different forms of the same email account. This means that you can register again and again, when each trial ends, to get a new trial — all with the same email account. Developers can’t do much to restrict this. Continue reading to figure out why.

Hack# 2: Many domains can refer to the same account

If you own any @gmail.com address, you can also use any @googlemail.com address for sending or receiving emails. Most websites and web services treat these email addresses as different. This allows you to register multiple accounts using both forms of a single email address.

Hack #3: Email addresses can be case-sensitive

Many people know that it doesn’t typically matter whether you write to letgrandmaseat@gmail.com or LetGrandmasEat@gmail.com.

But it turns out that the part before the @ can be case-sensitive. This is because it is entirely under the control of the host system. According to RFC 5321, section 2.3.11:

“The standard mailbox naming convention is defined to be “local-part@domain;” contemporary usage permits a much broader set of applications than simple “user names.” Consequently, and due to a long history of problems when intermediate hosts have attempted to optimize transport by modifying them, the local-part MUST be interpreted and assigned semantics only by the host specified in the domain part of the address.”

So, if you use a luxury domain instead of a well-known one (like gmail.com), it’s best to keep the address in its original case. But if you accidentally mistyped and the resulting address does not exist, you will likely receive an email telling you that your message was not delivered.

These naming conventions completely depend on the host systems. Technically, anyone who can create email servers can set up any naming rules they want. They can use different domains they own for the same mailbox, allow or deny any of the valid characters in the email addresses, and so on.

Hack #4: “Disposable email” services

There are a couple of services which allow you to create a temporary, disposable email address for free. These exist to allow you to access information when you don’t want to register because of privacy concerns or other issues.

And these services are used by many people. Just check how often test@mailinator.com receives emails, and how much uninteresting junk there is.

To prevent the use of fake emails, some services use different email validation solutions. And yes, they help in some cases. It becomes more difficult to create a bunch of email addresses which will bypass validity checks. Lots of robots and spamming programs will die here, which is why these services exist.

Do email validation services sound like the solution to fraud or spam problems? In most cases — yes. But not for all possible problems, as you’ll read next.

Hack #5: You Can Create Almost Any Email Address For Free

When you own a domain, you can set up an email service for it. There are a couple services which let you do this for free. And there are a couple providers which provide free domain names for everyone.

If you can follow instructions to set up a domain and an email on it, you can create any.name@any.domain.XY email address for free. The detailed instructions to set up such an email are beyond of the scope of this article. But in short, the procedure looks like this:

1. Register a free or paid domain by filling out a typical registration form.
2. Select one of the “email for domain” providers (for example, Zoho), and follow the instructions to create an email account for your domain.

Yes, it’s that easy.

The implications for email users

By adding special symbols like dots or plus signs to your email address (for some email providers) you can get the most of it. You can manage incoming group emails and register multiple times on some sites. And it lets you put different forms of your address on your business card so you know the source of incoming emails.

Disposable emails, as well as temporary phone number services, can protect your privacy when you need to register on an unwanted website for any reason.

And finally, nowadays anyone can register a free domain name and set up an email on it. This gives you a unique, personalized, and valid email address.

The implications for developers

In short, there is no way to be completely sure that we can trust a given email. But these are some practices that can reduce abnormal use of an email address:

1. Do you store emails as primary IDs in the database, but want to disallow users from registering on your service multiple times? If so, you can use a library like this to normalize email addresses before storing them as IDs. This library removes dots and plus-sign endings according to different well-known email providers, changes the address to lowercase, and fixes domain names to a single form (googlemail.com → gmail.com).

2. If you use libraries that normalize email addresses, let users keep their email address in the form it was originally entered. For example, if a user registers on your service with username+bank@googlemail.com, make sure that all the emails are sent only to that address. Don’t send to the “normalized” username@gmail.com address.

3. To detect disposable email addresses, you can use open-source libraries like this. You can also rely on the services which provide APIs to validate email addresses. There are more than 2,000 domains in the list!

I hope you learned something new by reading this article. If you like it, let it spread over Medium by pressing on the “clap” button as many times as you will ? .

Thank you for reading!

TL;DR (updated 2021) You can now use dataunlocker.com service, which is a fully managed solution for fixing ad blockers' impact on the client-side analytics tools such as Google Analytics. Log in to DataUnlocker Admin to complete the quick 3-step setup of your web application or website.

The article below from 2017 explains some principles used behind the solution, as well as it is described in the solution's readme.

When your product is just getting started out, every single user matters. So does the data about how they interact with your product.

If you’ve tried using analytics solutions like Google Analytics, you may have faced an issue where your analytics collection was blocked by ad blockers.

According to PageFair, up to 30% of Internet users use ad blockers in 2017, and this number is constantly growing.

This article will explain some technical approaches you can take to prevent ad blockers from also blocking your analytics. We’ll use Google Analytics in this article, though much of this could be applied to other analytics tools.

Some ways you can bypass ad blockers

Almost all ad blockers work through the same methods: they disallow some http(s) browser requests for content at URLs that match a certain mask from their filtering base.

Most ad blockers blacklist www.google-analytics.com by default, and block any attempts by the Google Analytics JavaScript library to send or retrieve the data from its analytics servers.

Luckily for developers, ad blockers don’t block requests to our own domain names by default, because doing this may hurt the web application’s functionality. This gap reveals a way to avoid analytics blocking until your web service become well-known enough for some of its URLs appear in ad blocker filters.

In fact, even after some URLs appear in the content filtering base, you can start playing with ad blockers by inventing terrible things, such as hourly changing analytics URLs (though this is beyond the scope of this article). Some of these approaches are applied by services like DataUnlocker.com and Adtoniq, which offer users adblocker-free experience even when ad blockers are turned on.

A high-level explanation of what we’re going to do

In this article, we’ll assume that we have no permission restrictions on the server side. We will write the demo solution (a few lines of code) for the Node.js platform. Once you understand how this works, you should be able to port this solution to any programming language or platform.

The solution I’ll describe is pretty minimal, and if you’re an experienced web developer, it may only take you a few minutes to put it in place.

We’re going to use a simple proxying approach without the need to diving into the Google Analytics measurement protocol. In short, the solution looks as following:

1. First, download the Google Analytics JavaScript library itself and host it on your server.
2. Then alter the code in the downloaded library to change the target host from www.google-analytics.com to your own domain name using find-replace.
3. Replace the link from the default Google Analytics script in your codebase to modified one.
4. Create a proxy endpoint to Google Analytics servers on your back end. One important step here is to additionally detect the client’s IP address and write it explicitly in requests to Google Analytics servers to preserve correct location detection.
5. Test the results. You’re done!

The full technical implementation walkthrough

All the code and described steps below are available on GitHub. The description below explains the method basics, and of course the suggested approach can be improved to be even more “anti-blocking.”

In Google Analytics, you start by acquiring a unique tracking ID for your property (web service). We will use UA-98253329–1 tracking ID in this article for the demonstration. Don’t forget to replace the tracking code to your one.

Google suggests adding this minified code to your web services to enable analytics:

<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script',
'https://www.google-analytics.com/analytics.js','ga');
  ga('create', 'UA-98253329-1', 'auto');
  ga('send', 'pageview');
</script>

In a few words, this code loads the Google Analytics JavaScript library if it wasn’t loaded before by inserting the script tag to the document. This library includes all the logic of analytics collection, and it is the only thing we need to proceed.

Step 1: Download and patch Google’s analytics library

Download the script directly from https://www.google-analytics.com/analytics.js, open it with any text editor and replace all occurrences of:

www.google-analytics.com

with this exact string:

"+location.host+"/analytics

By patching the analytics library in this way, it will start making requests to the local host (my.domain.com/analytics) endpoints instead of www.google-analytics.com. Place this patched analytics.js file on your server after the replacement.

Step 2: Replace the analytics script with the patched one

Let’s modify the Google Analytics embedding code in that way so it use our patched library instead of default one:

<script>
(function(i,s,o,r){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date()})(window,document,'script','ga');
  ga('create', 'UA-98253329-1', 'auto');
  ga('send', 'pageview');
</script>
<script src="/analytics.js" async></script>

Note that here browser will search for the patched analytics script in the document root of your server, in this case, my.domain.com/analytics.js. Check whether you put the script in the document root or changed the path in script tag above. You can also check the results by running a test on your local server (see the readme for how to run the GitHub example).

You should see something like this in browser’s developer tools:

Ultimately we want the act of downloading your patched analytics.js to return a successful response — a 200 (OK) or a 304 (not modified) status. But at this point, the request to my.domain.com/analytics/collect should respond with 404 status, since we haven’t implemented the proxy server yet.

Step 3: Implementing the Simplest Proxy Server

Now we’re going to code a bit. Our goal is to implement the proxy server, which will transport our analytics requests from our server to the real Google Analytics server. We can do this in many ways, but as an example, let’s use Node.js and Express.js with the express-http-proxy package.

Gathering all the files in the example together (see GitHub), we should end up with the following JavaScript server code:

var express = require("express"), 
    proxy = require("express-http-proxy"), app = express();

app.use(express.static(__dirname)); // serve static files from cwd

function getIpFromReq (req) { // get the client's IP address
    var bareIP = ":" + ((req.connection.socket && req.connection.socket.remoteAddress)
        || req.headers["x-forwarded-for"] || req.connection.remoteAddress || "");
    return (bareIP.match(/:([^:]+)$/) || [])[1] || "127.0.0.1";
}

// proxying requests from /analytics to www.google-analytics.com.
app.use("/analytics", proxy("www.google-analytics.com", {
    proxyReqPathResolver: function (req) {
        return req.url + (req.url.indexOf("?") === -1 ? "?" : "&")
            + "uip=" + encodeURIComponent(getIpFromReq(req));
    }
}));

app.listen(1280);
console.log("Web application ready on http://localhost:1280");

A few last lines here do the proxying. The only trick we do here is instead of just proxying, we detect and append the client’s IP address explicitly in a form of a measurement protocol URL parameter. This is required to collect precise location data, because all the requests to Google Analytic originally come from our server’s IP address, which remains constant.

After setting up our server proxy, we can check whether the request to our /collect endpoint now successfully returns a 200 OK HTTP status:

We can use, for example, an anonymous connection to verify that location is also picked up correctly.

This “proxy server” approach is a fast workaround for analytics that enables your services to avoid ad blockers. But this method relies on the browser side, and if the browser script for some reason does not send analytics information to our servers, we get nothing.

The last possible step if you want to implement a solid solution is to send analytics directly from the server by using server-side libraries available for different languages (NodeJS, Python, Ruby, PHP). This approach will definitely avoid any content blockers, as every request to analytics servers comes directly from our servers.

Again, the demo application is available on GitHub, feel free to test it! Let me know if you have any feedback or interesting experiences using this approach.

Thanks for reading!