I was hitting my limits. Everything I was doing I did decently fast, but I didn’t get any faster.

I've since learned that Vim is the only editor that you get faster using with time.

Vim is based solely on shortcuts. When I discovered that and played around a bit, I felt numb and a little stupid, having not learned the shortcuts (called Vim language) much earlier in my career.

I realized there was a keystroke to get to any specific position I wanted to jump. It was like a game, seeing if I could use fewer shortcuts to accomplish a particular edit. It’s where many Vim users get a lot of pleasure from coding and writing. It felt liberating, moving my cursor with the precision of a surgeon.

Although speed is a smaller benefit, it got me started when I saw others navigating in Vim. After climbing the steep learning curve, it’s still one of the most powerful skills I've ever learned in my career, working for a living on a computer.

Let’s debunk the myth of Vim and learn how it’s possible to remember all the shortcuts using the specific Vim language. We'll see how to move with vim motions, and I'll share what I’ve learned so far, and why you might give Vim a try as well.

Learning the Vim Language

Lots of things have been said about Vim – how fast it is, how only Linux nerds use it, and that it’s impossible to exit Vim.

For myself, I fell in love with the “Vim language”. You see, I’m bad at remembering anything and thought that Vim was not for me. But this wasn't the case for one specific reason: Vim motions and its language.

I learned that there’s a grammar behind the editor. With it, you express what you want to do first, how many times, and then what you want it to apply.

Let’s get deeper into Vim and the language behind it.

How the Vim Language and Motions Work

Vim has a terrific language or grammar behind its shortcuts. Instead of remembering a thousand shortcuts, you can learn a couple and combine them.

These are often called the Vim language or Vim motions for moving around. This has nothing to do with the editor yet – these are universal and available in other editors as well.

For example, there's VSVim for VSCode, IdeaVim for the JetBrains products, Vintage Mode for Sublime, and so on. But there are also Browser extensions like Vimium or Firenvim, and Gmail even adapted some of Vim's shortcuts for navigation (j, k for moving, g for jumping).

Everyone who types on a computer eight hours a day should learn the Vim language. Yes, it’s hard in the beginning, but that’s the case with everything new and different. But getting better every day and having more fun coding or writing should be motivation enough. You're not too busy to learn - you'll learn as you go.

Vim Grammar

Just as spoken language grammar has verbs, subjects, and objects, so does the Vim language. The grammar has different verbs to begin with. Copying (or yanking) in Vim with y, deleting with d, pasting with p, changing with c, and so on.

For example, the easiest shortcut is copying a line with yy. In this case, yank is the verb and the second y is a synonym for y_. The y is doubled up which makes it easier to type since it's a joint operation.

Next, we can add movements. Each verb takes a subject for their movements. There are lots of movements (more in the next section) – the easiest is with numbers.

For example, to copy three lines, you add a 3 in front, such as 3yy. You can do that with all verbs, like deleting three lines is 3dd. Another would be { and } to move to the beginning or end of the paragraph, respectively.

In addition to verbs and subjects, the Vim language also has objects. For example, we can save text into different clipboards (called a register in Vim) with "ay. Here, we copy it into register a, which would be the object. We can paste it again by doing the same but using the verb paste instead of yank "ap.

There are even adjectives and adverbs with prefixes. Usually, you use a verb and an object. But instead of going down three lines with 3J, which joins the following three lines, you could add d5}, which means “delete from the current line through the end of the fifth paragraph down from here.”

For me, the most magical thing about Vim is how you navigate and edit text – and it still has nothing to do with the editor.

Sure Vim was the one that introduced and perfected these actions, but again – you can get them anywhere else. This goes deep into the Vim language, yet we still need to touch on the editor. This is important to know.

I hope you've started seeing the power of such patterns, though. With a couple of verbs and objects, you can already know hundreds of combinations without memorizing each one individually.

You can watch a video on Mastering the Vim Language or read a full exposition of the Vim language on this terrific StackOverflow comment.

Vim Motions

Vim motions are how you navigate, whether you navigate to the end of the word or back to the start of the document – these are all motions.

These are the first things you start learning (and hating) when learning Vim. They're extra hard to figure out initially, but they're something you'll want to use everywhere when you get used to them.

Instead of using arrow keys, Vim uses jk to move down and up and hl to move left and right. The main idea is to use the keys your right hand naturally rests on. You do not need to move your hands or even fingers for navigation.

Again, this seems like a small thing, but once you've learned it, you know why everyone is telling you about it.

Some common motions are:

h,j,k,l - left, down, up, right
w,W - to start of next word or WORD
b,B - to start of previous word or WORD
e,E - to end of word or WORD
$   - to end of line
^   - to start of line

You can find the most important motions to start with in this cheatsheet:

This Vim Command Cheat Sheet is from Cloud Guru.

How to Use Vim Modes (normal, insert, visual, and command)

Modes are another thing that might get you confused at the beginning.

When you launch Vim, you are not typing what you click on your keyboard as you are not in the “insert” mode that you're likely familiar with from other editors. Instead, the normal mode you are in lets you do the commands explained in the above Vim language and motions.

Vim is the only editor that optimizes editing text instead of writing from a blank page.

That’s another reason Vim makes you so efficient: you have different modes for each phase of your current work or task.

• Normal mode is for reading code and navigating quickly.

• Insert mode is for when you want to add some code or text.

• Visual mode is unique, the same as highlighting text with the mouse, but with the above Vim motions.

• And command mode is the powerhouse, where you can type Linux commands such as formatting a JSON file with :%!jq (whereas jq is a command line tool installed on your machine) and execute them within Vim. This is also where you can use Vim commands such as :sort for sorting your files.

I could go on here, but I want to dive into the editor itself now and explore why I learned it initially and how to get started.

Introduction to Vim the Editor (Neovim, Lunarvim, and Helix)

So what is Vim the editor, then? It started with the simple vi editor, a basic editor that implements the Vim language and can edit text. It's a little like Notepad++, which you might use on Windows, but without a mouse and context menu.

Vim is simply an improved version of Vi with more features.

Today there is even a newer version of Vim called Neovim. This version is super popular, and I’ve started using Neovim as well. Compared to Vim, Neovim uses Lua, an actual programming language, to configure and extend the editor. This makes writing plugins and configuring Neovim easier compared to Vim's native Vimscript.

Neovim is a great place to start learning Vim today, as it has so many awesome plugins. Neovim also won the most loved IDE on the StackOverflow survey a couple of times, last in 2022.

There is also an editor called Helix built in Rust, but it has minor deviations from the Vim language, which make it a less optimal place to start.

If you want to get started without needing to know anything about Neovim and spending hours on configurations, you can begin with LunarVim. It's a distro with all the features you know from VS Code already included.

Suppose you are comfortable with the terminal and realize you want to change the editor to your liking. In that case, you can kickstart your journey with a simple single-file configuration with many explanations that will work out of the box. You can also learn each config by opening the single config file.

Why I Learned Vim

Using the standard input method we use in our editors daily, we will eventually stagnate at a certain level. Sure, you can use cmd+arrow-keys (on a Mac) to jump to the beginning of a line or option+arrow-keys to jump between words instead of characters.

But what happens once you've mastered that? What if you need to change something in the middle of a sentence? There is no other way to jump several times with this option, or you move your hands away each time to reach the mouse to click on the exact spot.

One day, I saw a coworker work in Vim, and everything clicked. The Vim language and motions were the things I needed all along. So I installed the VS Code plugin, watched a couple of YouTube videos, and started my journey to learn the basic movements.

I also love learning new things and, even better, I'm always looking for ways to make me more productive 😉.

But as many of you might have experienced, the hardest part of learning Vim is getting started. The initial learning curve is very steep. Below is an illustration that shows this :).

It took me also two or three times trying to start learning Vim until I fully grasped it. I switched back and forth. As well as having to get work done, it is sometimes hard to switch entirely from one day to another. But I just loved learning all the movements, and I knew it would make me faster after a short time.

Why I Love Vim

I have been using Vim for only eight months, and coding with Python for around six years. I've been using other code editors since my career started in 2003. Each editor I used had its strengths and its appeal. But I never experienced such efficiency gains as I have with Vim.

In the end, use the editor that works best for you. Personally, I want my editor to help me work as fast as possible, especially since I use it daily. Investing a bunch of time learning Vim is necessary, but it pays off over time. That’s the whole point with Vim and especially the Vim language.

An underrated skill in general among programmers is using the terminal. By learning your editor, especially with Vim, you will naturally learn more about the terminal and improve your Linux skills (reverse search, lazy git, Tmux, and many more).

Before Vim, I only used the terminal if I had to. I googled everything, and today, I use the terminal with its helpful manuals whenever I can.

Sometimes I’m surprised by myself as well, and it’s super nerdy – but it’s so effective. I’ve become a much better developer since starting to get comfortable with Vim.

Tweaking and optimizing Vim can take hours and days, and it's unavoidable in the beginning. But after a while, your dotfiles mature, and you start changing things less. You will also get much faster at trying out a new plugin or adding a remap.

Also, Vim is fun! Working in Neovim is one of the highlights of my everyday work. Improving your text editor and making it your own – maybe in ways no one else has optimized – is awesome.

For example, I write a lot, so I optimized for writing markdown and programming in Python. That’s what adds a lot to my happiness as a coder.

Because of all this, TJ DeVries calls Neovim a PDE (Personalized Development Environment), not “just” an IDE. You can learn more about this in ThePrimeagen’s truly inspiring Vim videos and learn why he used Vim in 2022.

Vim also manifested minimalism more in me. I used the terminal instead of fancy GUIs and plain text files for clarity, freedom, blazingly fast shortcuts, no vendor lock-in, and staying in the Flow with the content in front.

Vim changed not only my workflow but how I was able to edit at the speed of thought. Instead of thinking, “I want to edit that word”, my fingers jump to that word and change it with a few keystrokes.

Vim for Data Engineering

My data engineering workflow uses Neovim with the LSP (Language Server Protocol) pyright installed with mason. There's much more with Tmux, but you can find all the details on dotfiles/nvim.

Vim for Writing

I’m still using Obsidian a lot more for writing (see more on my PKM workflow) due to its additional features of supported images, backlinks, graphs, and plugins specified for note-taking, such as ReadWise (syncing my highlight from books, and tweets), Dataview (using notes as a database), Excalidraw (drawing with Markdown format), Templates, and so on.

Nevertheless, I write more and more in Neovim. For now, I use ZenMode (for centering the text), Grammarly (for linting grammar), write-good (linting grammar), and specifically Obsidian.nvim (follow backlinks, and so on.). You find all details in my dotfiles.

In Obsidian, I use the Vim mode Obsidian .vimrc and map most hotkeys to my Vim settings. At the same time, I've been writing more and more in Neovim and have been progressively moving over to full Neovim.

Plugins such as Telescope and simple grep features that I use for coding work very well with Markdown. Here are some clips showing what is possible in an excellent talk about Writing, Editing, and World-Building at the speed of thought with Vim.

Why You Should Learn Vim, Too

When I heard about Vim, I thought that it was only for software engineers and Linux nerds 😅. I never thought I was going to use it as well. But how did I get into it?

I've already shared some reasons why I love Vim. But it really changed all my workflows, not only as a developer but also how I surf the internet, write, navigate, and use tools. I search for a Vim mode in any application I use.

But if you don't enjoy fiddling and optimizing your workflow, and if you don't write or code for a living, Vim might not be for you. Start with your current editor and activate the Vim mode before you do anything with Vim. It will save you a lot of frustration, trust me 😅.

How to Get Started Using Vim

There are many good resources that'll help you get started with Vim. The easiest is to type vimtutor into your terminal, which is an interactive tutorial for Vim.

I've already linked a few YouTube videos above – especially check out ThePrimagen starting with Vim As Your Editor or Why even bother with Vim or Neovim in 2022.

I started with Ben Awads’s Vim tutorial back then. An excellent Lecture: Editors (Vim) (2020). Mastering the Vim Language. I also collect a small playlist on YouTube with Vim content. A big inspiration also dev workflow using Tmux and Vim from Takuya.

Wrapping Up

We have learned that Vim is a powerful text editor popular among developers. It's based on shortcuts, called the Vim language, which can make coding and writing faster and more efficient.

With Vim, you can jump to any specific text position and rapidly make precise edits. While learning Vim can be challenging, it is well worth the effort in the long run as it will improve your productivity and bring joy to your coding experience.

If you want to go further, try Tmux, which plays well with Vim. You could even go one level deeper, which is a dedicated keyboard layout such as Dvorak or Halmak (which I started learning at some point). Or buy a fancy ergonomic keyboard or build one yourself.

Thanks for reading this far. I hope you enjoyed this article. I'm looking forward to hearing your comments and experiences.

You can read more of my other tutorials at www.sspaeti.com.

I asked myself isn’t there a smarter way to communicate (discuss, argue, debate) at work? This is what I’m going through in this article.

Why email is so popular

Let’s start with why email is so popular and still used as the number one communication in the vast majority of companies.

Some strengths and big advantages email has for sure is sharing fast and in a direct way to a number of people. It is location independent. A very handy feature, it keeps a record of all the communication and you can always go back to it.

The costs are obviously very low. And one big benefit it’s still the best way to market your target users to make an actual sale.

The bad things about email

Although one might argue email is efficient and productive communication, I’d argue against it. It’s ineffective because it’s based on old fashioned expectations. We approach email like a letter, where every email will be answered immediately the moment the letter arrives.

A major problem that email has is distraction, which can be seen as an open to-do-list for everyone else to put a task on top of it. There are different studies ( The Cost of Interrupted Work, Disruption and Recovery of Computing Tasks), one says that workers are switching tasks every three minutes, mostly due to email. Around 36 (!!) times an hour.

These interruptions are particularly bad in case you want to get tasks done. Normally every interruption needs around 20 minutes to get back on focus where you were just before the interruption. As productivity is best in bigger junks it’s even more important to block out email or have other ways of communications with uninterrupted timeslots.

“Professional activities performed in a state of distraction-free concentration that pushes your cognitive capabilities to their limit. These efforts create new value, improve your skill, and are hard to replicate.” Cal Newport, author and blogger who writes about the concept of deep work.

For Newport, the benefits of deep work are continuous improvement of the value of your work output, an increase in the total quantity of valuable output you produce, and a deeper satisfaction at work.

Besides hindering your ability to accomplish the essential aspects of your workday, email also contributes to the overall stress. I’d even go so far to say that relationships at work are hindered through email. A study of the University of California Irvine confirms that and says: “Without email, our informants focused longer on their tasks, multitasked less, and had lower stress.”

Above all these disadvantages, maybe one of the biggest is the lack of structure. If you have emails, everything goes down one rabbit hole. Every person needs to organise their emails themselves. Only these tasks can take a lot of time which could be spent much more wisely on project tasks.

On top of the missing structure is the lost history if an employee is leaving the company. Many very helpful and informative emails, and the work in organising them, are lost.

Spending less time on email

Measuring your time is a good way to change how you use email. I did exactly that with a tool called RescueTime. This way you can clearly see how much time you spent on each and every app. Overall I used 24% of my time for communications — this included WhatsApp, Facebook, my friends. However, only at work, I found myself using 1 hour and more every day on my email client. Sure some of it is necessary, and some are even for scheduling meetings. But still it was too much, and I’ve tried to keep it less and use email more efficiently since then.

However, some of the time spent is not due to my behaviour. It is the company and it depends on how your team or company communicates. Some studies say the average office workers receive 110 messages a day and spend 28 percentage of their time handling email. That translates into 13 hours a week or 650 hours a year of e-mail checking. Imagine how powerful it would be for your organisation to reclaim that lost time!

There are jobs where emailing is your primary work, like working in a call centre. In that case, you cannot change that or benefit immediately. But what about managers who spend between five and 20 hours a week reading and writing emails, as this article in Forbes says. Or even more for people in tech, where their main purpose is to develop new features, solve problems and be creative? In these cases, email is just not what they should do to be successful in their work.

What is the need of today’s communications?

Let’s find out what are our requirements for emailing, or more generally, communication. Besides knowing which users are using which communications, often it’s not only a technical adjustment but more a change in culture.

As Millennials are now the largest group (in the U.S. labour force), we should also make sure we use ways and tools that suit them. Millennials have grown up with technology and are more familiar with newer communications styles in Social Media. And as email was first used in 1965, I would say it’s not the best fit for today’s needs anymore.

Characteristics of Millennials

Let’s dig a little deeper into what the attributes of a millennial are. They like to be in a community and to be part of something, given a real purpose. They use their mobile phones all the time. They are not so severe all the time, they appreciate honesty, and they want to be heard. They want to engage with top-level leaders, give feedback and constructive criticism, and they want success for everyone involved, short and to the point. Also, they are into new tools and prefer written communication.

Managing an important project or handling complex tasks can be overwhelming. Most millennials would rather leave the telephone untouched and streamline their workload using written communication methods. Sending out a message via instant messaging is a quick, effective way to get in touch with someone. Written communication through a social network also enables millennials to assign a project without losing any details — and all of the information can be recalled later. This helps when keeping track of project updates because everyone involved is accountable. See more about how to work with millennials here and how to do that effectively.

How to turn around communication? What can we do

To fit the new generations, we should use tools that help achieve their goals.

Tools and the way of working

One of the most important is the tool or platform you are using. If you’re still using a waterfall methodology and Microsoft Word for specifications, only one person at a time can edit it and it makes it hard to share. That way you don’t allow feedback and changes to your ideas. On the other hand, in methodologies that support a feedback culture e.g. like Scrum or similar, you want to have improvements and feedback every two weeks (or as long as your Sprint is).

One feature of this platform or tool you want to use must be that everything is in the same place. Discussions, Meeting Calendar, new idea proposal, status updates, file-share, chats, even client-access. Maybe email forwards and uploads important emails coming in on the side. But reports, search, notifications and most important, the ToDos/Defect/Stories should all be in one single place.

Or if not available, use the powerful integration that most tools have. As the tools chain is rising every day, it’s important to integrate certain important tools in your communication and management platform as well. If you have a git-repository or any file-share you can add extensions that integrate into your tool.

This way you minimise additional communications and coordination as you don’t need to synchronise tasks, statuses, open discussions etc., and everyone sees it all in one place. You get immediate feedback and everyone can contribute. The feedback is specific to a task, idea, defect etc. and right where it belongs.

The To, CC, and BCC fields

Targeting a large group in email is a mess, especially as you need to decide up front who to put in the “To”, “CC” (carbon copy) and “BCC” (blind carbon copy) fields. It’s easy to forget someone, or even worse, you add everyone who comes to mind. Try to target only the people needed.

Start with a small group and only expand when the idea or the story gets more mature. If you want everyone to see it, you add just the core team and keep the access-rights open for everyone. That way, nobody is left out but they don’t get a ton of unnecessary notifications. Plus they can only have a look when you have time or want to see it.

As you can create email rules for every topic and sender, it’s somewhat cumbersome. Especially in the above-mentioned tools, you can specify notification for each project, each discussion, each task. That way you decide when and how you get notified. Where in email, you get random email notifications, unstructured.

Offices and their noise-level

Besides the platform and tool, it’s critical to have a productive working environment. As more companies are changing to an open office, as it’s helpful to have everyone in one room, it is contradictory to productive, deep work. As you get interrupted every time someone is on the phone, having an important discussion that does not include you, etc. Your brain is wired to listen to it, even though you don’t want to — you will always find yourself listen to it.

You can use headphones with noise cancelling and music to focus, which makes working way more productive, but you still would be more productive in a quiet environment. I like the definition of your office as a library like J. Fried and D. H. Hansson describe in their book It Doesn’t Have to Be Crazy at Work. Because if you go to a library you expect everyone to be quiet and work productively. Shouldn’t that also be the way an office is?

Same goes for meetings. When the office is a quiet place, then you need to have many discussions and workshops in meeting-rooms. But I would distinguish between meetings and productive working together. As productive one-on-one workshops to solve problems certainly increase productivity (a lot!), long status updates or unproductive discussions, if they’re not solution driven, can go the other direction.

Think of meetings as if they cost you something, as Seth Goldin said in his article about Making meetings more expensive. You would probably think twice if and how to have your meeting:

1. whom to invite

2. making meetings start right on time

3. ensuring that every meeting has a clearly defined purpose

4. managing the flow of information, including agendas and coordinating the discussion

5. issuing a follow-up memo to everyone who attended the meeting

Agile and remote work

As mentioned earlier, I would argue that you should work in an agile way. It works hand-in-hand with most of the points covered above. If you work agile, you use scrum or similar methods with well-defined user-stories that make them easy to discuss with your stakeholders.

Define tasks for your developers. This way you need less synchronisation as tools allow you to move a user-story if needed easily. Additionally, you see what everyone is doing, where they might get stuck, and you can plan your sprints with ease and also make remote work easy.

What are the tools to do that?

There are many tools out there. Here are some, but of course this list is not complete:

• Basecamp — maybe the most completed
  Managing projects, working with clients, coordinating with contractors, and communicating with your team

• Slack — very good for a one-to-one replacement for email in projects or teams
  A powerful tool for making team communication fast and easy.

• Gitlab — best option very well working with git commits
  Single application for the entire software development lifecycle. From project planning and source code management to CI/CD, monitoring, and security.

• Asana — maybe best free option
  Work management platform teams use to stay focused on the goals, projects, and daily tasks

• JIRA — especially good for development with project, bugs and tracking
  Issue tracking product that allows bug tracking and agile project management

• Trello — for small projects and small amount of people, very lean
  Boards, lists, and cards enable you to organize and prioritize your projects in a fun, flexible, and rewarding way.

• Taiga — open-source basecamp
  Project management platform for agile developers & designers and project managers

Some less famous email replacement tools (in alphabetic order):

• activecollab.com

• clickup.com

• copperproject.com

• freedcamp.com

• glip.com

• huddle.com

• mavenlink.com

• monday.com

• nostromo.io

• nozbe.com

• php-collab.org

• proofhub.com

• proworkflow.com

• teamwork.com

• wrike.com

It also comes down to how your company communicates: journalists, for example, tend to constantly ping each other with ideas and very short questions, and thus Slack might be the best fit.

There are also even more radical tools where all communication and processes are based on business workflows. An example is Tigerconnect.

Conclusion

Whatever you decide, whether you still using email or a new, more modern communication method, keep in mind the time spend on communication. Make it efficient and not distracting. Also, write with a cause. Don’t try to be the first one to answer. Better think about it thoroughly and reply only if it adds value to the conversation. Moreover, use if-then-else in your answer to foresee where the conversation might go to avoid back and forth communication.

Define clearly when to use which communication channel and what you want to achieve in order to avoid too many channels.

Lastly, with the right tool, it’s actually fun to communicate in an efficient way, solving issues and user-stories together and reducing weekly meetings to a bare minimum. No more missing deadlines, no more confusion about “I thought so and so was responsible for that”, and being productive as never before.

Related Links:

• How and why to use discussion threads instead of chats, emails or meetings — Basecamp

• Eight reasons why you should replace email with management software

• Scientific Reasons Why Email Is the Absolute Worst

• Replacing boring status meetings or status updates — Basecamp

• How quitting email helped my company communicate better

• The key to better work? E-mail less, flow more

• 10 Straightforward Ways to Improve Communication at Work

• As an example of importance, Facebook even bought a whole platform recently: Facebook acquired Redkix (unified communication)

Originally published at https://www.sspaeti.com on April 25, 2019.

For this post, I chose some open-source technologies and used them together to build a full data architecture for a Data Warehouse system.

I went with Apache Druid for data storage, Apache Superset for querying, and Apache Airflow as a task orchestrator.

Druid — the data store

Druid is an open-source, column-oriented, distributed data store written in Java. It’s designed to quickly ingest massive quantities of event data, and provide low-latency queries on top of the data.

Why use Druid?

Druid has many key features, including sub-second OLAP queries, real-time streaming ingestion, scalability, and cost effectiveness.

With the comparison of modern OLAP Technologies in mind, I chose Druid over ClickHouse, Pinot and Apache Kylin. Recently, Microsoft announced they will add Druid to their Azure HDInsight 4.0.

Why not Druid?

Carter Shanklin wrote a detailed post about Druid’s limitations at Horthonwork.com. The main issue is with its support for SQL joins, and advanced SQL capabilities.

The Architecture of Druid

Druid is scalable due to its cluster architecture. You have three different node types — the Middle-Manager-Node, the Historical Node and the Broker.

The great thing is that you can add as many nodes as you want in the specific area that fits best for you. If you have many queries to run, you can add more Brokers. Or, if a lot of data needs to be batch-ingested, you would add middle managers and so on.

A simple architecture is shown below. You can read more about Druid’s design here.

Apache Superset — the UI

The easiest way to query against Druid is through a lightweight, open-source tool called Apache Superset.

It is easy to use and has all common chart types like Bubble Chart, Word Count, Heatmaps, Boxplot and many more.

Druid provides a Rest-API, and in the newest version also a SQL Query API. This makes it easy to use with any tool, whether it is standard SQL, any existing BI-tool or a custom application.

Apache Airflow — the Orchestrator

As mentioned in Orchestrators — Scheduling and monitor workflows, this is one of the most critical decisions.

In the past, ETL tools like Microsoft SQL Server Integration Services (SSIS) and others were widely used. They were where your data transformation, cleaning and normalisation took place.

In more modern architectures, these tools aren’t enough anymore.

Moreover, code and data transformation logic are much more valuable to other data-savvy people in the company.

I highly recommend you read a blog post from Maxime Beauchemin about Functional Data Engineering — a modern paradigm for batch data processing. This goes much deeper into how modern data pipelines should be.

Also, consider the read of The Downfall of the Data Engineer where Max explains about the breaking “data silo” and much more.

Why use Airflow?

Apache Airflow is a very popular tool for this task orchestration. Airflow is written in Python. Tasks are written as Directed Acyclic Graphs (DAGs). These are also written in Python.

Instead of encapsulating your critical transformation logic somewhere in a tool, you place it where it belongs to inside the Orchestrator.

Another advantage is using plain Python. There is no need to encapsulate other dependencies or requirements, like fetching from an FTP, copying data from A to B, writing a batch-file. You do that and everything else in the same place.

Features of Airflow

Moreover, you get a fully functional overview of all current tasks in one place.

More relevant features of Airflow are that you write workflows as if you are writing programs. External jobs like Databricks, Spark, etc. are no problems.

Job testing goes through Airflow itself. That includes passing parameters to other jobs downstream or verifing what is running on Airflow and seeing the actual code. The log files and other meta-data are accessible through the web GUI.

(Re)run only on parts of the workflow and dependent tasks is a crucial feature which comes out of the box when you create your workflows with Airflow. The jobs/tasks are run in a context, the scheduler passes in the necessary details plus the work gets distributed across your cluster at the task level, not at the DAG level.

For many more feature visit the full list.

ETL with Apache Airflow

If you want to start with Apache Airflow as your new ETL-tool, please start with this ETL best practices with Airflow shared with you. It has simple ETL-examples, with plain SQL, with HIVE, with Data Vault, Data Vault 2, and Data Vault with Big Data processes. It gives you an excellent overview of what’s possible and also how you would approach it.

At the same time, there is a Docker container that you can use, meaning you don’t even have to set-up any infrastructure. You can pull the container from here.

For the GitHub-repo follow the link on etl-with-airflow.

Conclusion

If you’re searching for open-source data architecture, you cannot ignore Druid for speedy OLAP responses, Apache Airflow as an orchestrator that keeps your data lineage and schedules in line, plus an easy to use dashboard tool like Apache Superset.

My experience so far is that Druid is bloody fast and a perfect fit for OLAP cube replacements in a traditional way, but still needs a more relaxed startup to install clusters, ingest data, view logs etc. If you need that, have a look at Impy which was created by the founders of Druid. It creates all the services around Druid that you need. Unfortunately, though, it’s not open-source.

Apache Airflow and its features as an orchestrator are something which has not happened much yet in traditional Business Intelligence environments. I believe this change comes very naturally when you start using open-source and more new technologies.

And Apache Superset is an easy and fast way to be up and running and showing data from Druid. There for better tools like Tableau, etc., but not for free. That’s why Superset fits well in the ecosystem if you’re already using the above open-source technologies. But as an enterprise company, you might want to spend some money in that category because that is what the users can see at the end of the day.

Related Links:

• Understanding Apache Airflow’s key concepts

• How Druid enables analytics at Airbnb

• Google launches Cloud Composer, a new workflow automation tool for developers

• A fully managed workflow orchestration service built on Apache Airflow

• Integrating Apache Airflow and Databricks: Building ETL pipelines with Apache Spark

• ETL with Apache Airflow

• What is Data Engineering and the future of Data Warehousing

• Imply — Managed Druid platform (closed-source)

• Ultra-fast OLAP Analytics with Apache Hive and Druid

Originally published at www.sspaeti.com on November 29, 2018.