Engineering @ PharmEasy

At PharmEasy, we strongly believe in open-source.

Our application backend is primarily in PHP with Yii 2 as the base framework. We try to utilize the extensions created by the great developer community and customize them according to our needs. We have started extending the parent classes of Yii and added our bits of code there. And when that’s not possible, we build the modules ourselves.

For example, one major drawback in PHP is the lack of threading and if you have a memory intensive process, chances are you’ll run into the infamous PHP – Fatal Error – Allowed memory size of <> bytes exhausted. In one of our internal application, we have a module that exports the data to a csv format which is used by our operations team. And as our data started to increase, one day we ran into the same memory limit issue. Now, we could have solved that by just increasing the memory_limit setting for that script, but that is not a good solution. And at PharmEasy, we always think of the long-term solution with scalability in mind, maintaining high code quality. So to overcome that, we built a generic background-job module which is a simple queuing system in PHP, to which we push all the asynchronous tasks. These jobs are then picked up by our worker and processed, based on the current number of jobs. Then we separated the export module, processing the data in batches to reduce the memory usage and pushed it as a background job. Now, we try to push most of the things as a background job, making our APIs and processes faster.

Background Jobs
Background Jobs

 

REAL-TIME APPLICATIONS

When it comes to real-time applications, nothing beats Node.js, due to its speed and efficiency. Many of our internal applications are built on Node.js using Socket.IO, and a customized version of log.io for real-time monitoring of the node apps. We try to make our node applications API based to achieve the Microservices architecture.

 

CACHE

We use memcached, and we use it heavily! We cache almost everything that we can and the data is distributed among 5 m1.small nodes. For invalidation, we use very VERY lightweight MySQL queries which tell us if we need to refresh the cache blob. This has helped us reduce our response times, as well as the load to our MySQL database, running on an Amazon RDS instance. Here goes our hit-miss ratio:

memcached - hit-miss ratio
memcached – hit-miss ratio

 

DATA ANALYSIS

We needed eyes in our systems, like the ones which can get deep into our code, application, process and give us insights. These could be to check the performance of a function, the errors in a system or even random counts of the API calls being made. We track EVERYTHING.

Everything is logged – every exception, every error, with their stack trace and other parameters. For that, we use MongoDB Cloud Manager to manage our sharded cluster of MongoDB, which is spread across 8 different Amazon EC2 instances. This helps us in keeping track of things.

We have combined different tools that make up a system that helps us collect and understand arbitrary data. We use statsd to collect the data, which is fed to a Graphite server and Grafana to build the dashboards from the data. We prefer statsd because it works on a UDP port, so we can just shoot the data and not care about the acknowledgement. One of our dashboards:

Grafana Dashboard
Grafana Dashboard

We are also testing a free version of Splunk Light, that receives data from universal forwarders installed on all the application servers, which helps us in search, analysis and real-time monitoring of the logs. This is how our real time error tracking dashboard looks like:

Splunk - Error Dashboard
Splunk – Error Dashboard

However, since we are inclined towards open-source products, we have to try the ELK stack (ElasticSearchLogstashKibana), as it looks promising, and it’s free 😀

 

DEVELOPMENT

We use Bitbucket to host our git repositories. From the start, we have streamlined the development process to make it as fast and productive as possible. We follow the Gitflow workflow, where each developer starts a feature as a new branch and starts committing in that branch. Once the code is ready for production, the developer creates a pull-request which is then verified by the reviewer and merged to a develop branch. The code is then tested on a staging server, and when ready, is released for production.

You probably must have heard this everywhere, but yes, Testing is very important! And we (like all other good developers :D) ensure that most of our code is covered by unit / functional tests. We use SonarQube as our code quality management tool, which helps in improving the quality of our codebase and view the test results and the code coverage.

Jenkins is our tool of choice for our Continuous Integration. It is also integrated with SonarQube and Bitbucket to provide automated results. We have created WebHooks in Bitbucket for Jenkins and Slack. So whenever a piece of code is pushed, a Jenkins build is triggered which:

  1. Executes all the tests.
  2. Generates the test and coverage reports.
  3. Executes SonarQube build which takes the reports as input
  4. Reports the status back to Bitbucket using the recently launched build status API.
  5. Sends a status message in our Slack channel.

This way, the code-reviewer as well as the developer can instantly know that everything is fine.

 

Like what we do? Think we can improve somewhere? Is something else better than what we use? Share with us.