How can a customer control developers: important metrics and useful services

Quality control in the field of development is difficult for the customer, because the development product itself is a complex system with a large number of elements. There are tools for monitoring quality, but they are too “technical”, mostly focused on debugging rather than control, and do not take into account business indicators. In addition, not all contractors will be able to achieve project transparency.

Fortunately, there is a solution: in this article, we'll share specific recommendations on what metrics and services help keep the entire development process under control.

How to control an IT contractor

The success of each project is determined differently. For an online store, the financial result (profit, profitability) is important. When developing a B2B portal, we will monitor what proportion of contractors use it. The automated document processing system must provide a certain percentage of accuracy. And so on, the main thing is to agree “on the shore” what metrics will be the goal.

“At the same time, it is impossible to determine only the overall success of the project and not analyze the process. Why is that? Without understanding whether the development process is proceeding efficiently, it will be impossible to identify clear patterns and understand the true reasons for a particular state of indicators, assess whether IT specialists are to blame or contribute to this, and how much is due to the influence of other reasons. The conversion rate has fallen, but who is to blame: the marketers ran a bad campaign, the seasonal factor intervened, or did the contractor increase 500 errors on the server in the last release?” — Andrey Putin, kt.team CEO

In addition, without a high-quality process, it will be very difficult to obtain high-quality results.

The following metrics indicate the quality of the process:

• static analysis of the release (number of errors in the code, front-end; we recommend using the following standards and tools: PSR-2, ECGM2, ESLint, W3C, PageSpeed);
• JavaScript reports on the number of errors + LogReport;
  
• APM typical vs anomalies (these are tracers that show (in milliseconds) how each piece of code works (normally or with deviations), and where you can “fall” inside to analyze the causes of anomalies);
• 404, 500, etc. errors in dynamics;
• metrics that are unique to a particular CMS;
• release culture (release frequency, schedule compliance, number of bugs);
• audit results (they should be carried out regularly, for example, once a month).

These metrics can be combined under one concept: technical debt.

The problem is that most often the management of the customer company looks only at the final result and does not know about the actual technical condition of the project. The main reason for this is that contractors do not make quality reports, and the project turns out to be “opaque”. This means that the client does not know all the risks and cannot broadcast this to their investors. This leads to the fact that the project's business model does not take into account all restrictions, and the labor and financial costs of processing the project's technical debt are not planned in advance.

Technical debt

Technical debt is a metaphor that means all the problems accumulated in an IT project that can be fixed. Debt doesn't always mean poor developer performance. He usually appears when working under severe restrictions: either in terms of time or budget. And since there are always restrictions in business, there is always technical debt; it can be monitored, controlled, reduced, but it cannot be completely eliminated.

“For example, we worked with a large e-commerce client who simply didn't have time to pay for hosting on time, but the project was on fire, and we had to make an ugly solution for him on the cluster (rebuild it to a less “beautiful” Docker). We tell the customer: “There is a technical debt here, but we did it on time. Let's remake this decision when the peak season is over.” Another case: the code worked great, but the developers noticed performance anomalies. This means that a suboptimal solution was applied somewhere (a technical debt arose). Why is this happening? This means that there were time limits or budget restrictions. It is necessary to redo it”, — Andrey Putin, kt.team CEO

The main thing with technical debt is to digitize it (assess its scale) and plan to reduce it.

‍Work to digitize the quality of the project (technical debt) includes:

1. collecting performance parameters through apminvest.com, newrelic.com, AppDynamics; these parameters mean server response time and a collection of application traces over at least the last 24 hours;
2. the queue of tasks under development and solved to date;
3. monitoring the number of errors — from static analysis errors to server errors;
4. monitoring user errors and error messages from the call center (if there is a call center or technical support hotline);
5. control of the number of procedures (processes) followed.

Business indicators also help to digitize technical debt in addition:

• messages from operators (#ошибка #не_подтверждено #подтверждено) per day;
• cancelled orders per day;
  
• anomalies in orders per day (deviations from the norm in terms of time, the amount of the average check, etc.) );
  
• closed tasks per day.

There are special tools for monitoring and managing site performance: APM (application performance monitoring) systems.

APM services

The most famous APM system in the world is New Relic. It provides the following performance indicators:

• response time, throughput, error rate, etc. ;
  
  
• providing external services;
  
  
• the most time-consuming transactions;

• cross-application tracing;
  

• transaction failure;
  

• deployment analysis, history, and comparison

Its disadvantages are its high price and the inability to fine-tune (there is a “boxed” solution common to everyone).

“IN KT.team we use our own product that replaces New Relic and gives us more opportunities: ApminVest. This system is convenient not only for quality management, but also for auditing the technical condition of the site (including digitizing technical debt). Since the main goal of business is profit, it would be nice to immediately see the impact of technical indicators on finances, in real time, and as clearly as possible in the form of an understandable analytical panel and dashboard. This is also implemented in ApminVest,” — Andrey Putin, kt.team CEO

‍Examples of questions that ApminVest can answer

1. What is the actual server uptime? When are there problems with work speed? Do server drawdowns coincide with your audience's peak activity? How does this affect conversions?

‍2. PageSpeed score monitoring: how does download speed affect your company's revenue? What is the current degree of optimization of your project?

‍3. How do static analysis indicators, the number of log errors, data from Google Analytics or your CRM change after each release?

‍4. You can customize the dashboard settings for a specific request to take into account any relationship between technical and business indicators. This is very convenient and puts the work of IT specialists under the full objective control of management.

‍What does this give the customer?

• ‍Faster error detection.
• Improving the quality of development: reducing the number of errors, reducing technical debt, reducing the risks of poor site performance after updates.
• ‍Full project diagnostics for “chronic” errors and technical debt that prevent the platform from functioning effectively.
• ‍Monitoring the status of the project, quick notifications about what's important to know. Telegram creates a project chatbot, which receives messages about all errors, and groups errors by type.
• ‍A common language for setting goals, which is understandable to all employees and increases the efficiency of the team's work.
• ‍Develop at a quiet and efficient pace.

Final recommendations for monitoring development efficiency

The quality of development must be controlled in order to avoid losses and save resources. This is easy to do: there are special services for monitoring application performance (New Relic, ApminVest, and others). Use them in your projects.

By the way, all customers KT.team get access to ApminVest. The reason? This helps us do our job as efficiently as possible. Clients see that the project is transparent, and this increases their trust; developers see that the results of each action are visible and perform tasks more responsibly; the customer's technical team speaks the same language as the development team, because they agree on what metrics they will monitor and what values will be considered the norm.

To learn more about ApminVest, ask a question in the comments — we'll be sure to get back to you within a business day.