Development Environment Set-up for a Custom Elasticsearch Plugin

Ricardo Ocampo
6 min readDec 10, 2020

--

In the last few months, I have been spending most of my time creating a search engine using Elasticsearch. The team decided to go with this technology based on its simple REST APIs, distributed nature, speed, and scalability.

I started prototyping in Python. It was easy to port most of the implementation’s components to Elasticsearch; however, there is a custom ranking model that is not available in the similarity module. The complexity of the algorithm is high. It would take forever to apply it to all the retrieved documents in a search.

The rescore functionality was the solution to the problem. The rescore model is only applied to the top N documents that result from an initial query. The latter allows the user to define a window size that meets their requirements of response time, recall, and precision. The final decision was to create a custom plugin on top of the Elasticsearch rescore module.

The Challenges

The implementation of a custom plugin requires knowledge of how Elasticsearch works and its components. The initial tasks that helped to finish on time an initial proof of concept and followed by the deployment in production were:

  1. Define a set of tools to navigate through the code, highlight errors, autocorrect, code formatting, and debugging.
  2. Understand the classes and methods that form the plugin and the importance of each of them.

Considering the previous topics, I am splitting the information into two articles. In this article, I explain the process of selecting an IDE, and the configuration of the project to effectively work on it. In an additional blog post, I will explain the most important classes and methods of a custom plugin and how to modify them to create a simple rescore plugin.

The Tools

The main ingredients to efficiently build the plugin are an integrated development environment (IDE) and Docker.

IDE

The IDE that I used to develop the plugin is Visual Studio Code (VS Code). I liked the fact that it is an open-source editor, and there is a big community maintaining it. Another option is Intellij IDEA. There is a free…

--

--

Ricardo Ocampo

Data Scientist, I research and blog about machine learning in my spare time