Development Environment Set-up for a Custom Elasticsearch Plugin
In the last few months, I have been spending most of my time creating a search engine using Elasticsearch. The team decided to go with this technology based on its simple REST APIs, distributed nature, speed, and scalability.
I started prototyping in Python. It was easy to port most of the implementation’s components to Elasticsearch; however, there is a custom ranking model that is not available in the similarity module. The complexity of the algorithm is high. It would take forever to apply it to all the retrieved documents in a search.
The rescore functionality was the solution to the problem. The rescore model is only applied to the top N documents that result from an initial query. The latter allows the user to define a window size that meets their requirements of response time, recall, and precision. The final decision was to create a custom plugin on top of the Elasticsearch rescore module.
The Challenges
The implementation of a custom plugin requires knowledge of how Elasticsearch works and its components. The initial tasks that helped to finish on time an initial proof of concept and followed by the deployment in production were:
- Define a set of tools to navigate through the code, highlight errors, autocorrect, code…
