Medical/DNA sequencing software

About client:

SaaS solution for DNA sequencing application. 85 ppl staff, 62 of which are in product development and delivery. Location: Florida, US. Main markets: USA, Germany, Nordic countries.

Development team location: USA.

Client Challenges

1. Technical Debt – Due to a long list of technical debt, the company is actively hiring, but every new hire in the dev team adds complexity to management and communication. It seems that the easiest to manage stage for the product is when it starts, and the team consists of up to 10-12 handpicked developers, but the size of the backlog does not allow the company to stay on such a low headcount level.
2. The Paradox of Irreplaceability – The longer developer works on the project — the more irreplaceable he becomes*. But even this precious developer, who theoretically should be most valued for the team, is showing a lowering month-by-month performance. Partly because they know they are hardly replicable for the organization and partly because of the burnout superimposed on bad management habits in the company. This led to the fact that the pool of developers who are working in the company but are low in performance can’t be fired because only they know at least one critical feature of the project, and firing one of them will risk the stability of the feature or further development of it. The worst part of it is that developers start to acknowledge their unique knowledge, begin to protect it, and sometimes even damage the possibility of understanding it by other developers by writing the worst code that would take months for other developers to understand and refactor. *even low turnover washes out old players on the team, so after a while, you have only a few people possessing the history of the project and codebase, accumulated knowledge on the business side of the project (decision made, and why & how they are represented in the code).
3. Low performance per work timeframe: Planning is never aligned with the actual output even close. Every sprint, they have to put 30-40% of tasks back in the backlog or reprioritize them with what was planned for the upcoming sprint. Even considering that project managers understand that the team needs to spend some time together to sync and that some time will be wasted on friendly talking, they still expected the output to be 85% of what was scheduled, but it never happened. This frequently fired up conflicts between teams management and team members and between teams management and CTO, operational and sales/marketing management. The sales/marketing management conflict seemed to be the most damaging for the organization because the sales department promised new features to the clients and leads that would always be delivered with a delay. The planned 3-month delivery promise from the sales department was actually delivered in 7 months on average, if delivered at all. Considering the tight competition in a niche, this damaged the organization much more than any potential expenses on investment in the dev department they tried to do.
4. High hiring costs and a lack of skilled and motivated developers on the market which entailed and had severe consequences for the business, including delayed projects, reduced productivity, increased labor costs, poor quality work, and difficulty competing with other companies. Management, being smart, didn’t want to give up shares for hired professionals. Still, being a relatively young company (2 years on the market), it didn’t seem stable for potential candidates in the US market, so they were left with the choice – to compete with other tech startups on the price factor or hire mediocre developers but for the reasonable price on the market. They chose the money factor over savings, which helped them in many ways. Still, it costs the org quite a buck – 60% more in salaries budget compared to other well-established companies + extreme $40k onboarding cost (recruiting agencies fees + sometimes signup bonus for the candidates).

Discovery

The organization presented our services as an outside consultancy, insisted by investors, not to make people worry that they are reviewed for their mistakes or to decide on layoffs so that the company and we can get honest feedback, and of course, not lower team morale.

1. Interviews with project managers showed that almost every onboarded developer was complaining to managers or his colleagues about the number of calls and syncs he should not or is not actually involved in. Further discussion with senior developers working and ex-developers in the organization showed that out of 40 hours of the work week, around 6-7 hours is wasted on small talk and friendly discussions among the team of 20 developers on average. Around 11 hours a week is spent on calls and syncs, 5 of which are actually related to the developer involved. 30% up to this 18 hours is added as what we call fragmented time – non-focus time spent drinking a coffee, having a break to fill up the water bottle or take a snack, or something like this before and after each call to get back in focus mode. It showed that, on average, 23,5 hours a week out of 40 are non-focus. As a result, the real focus time on the work on its best weeks is a maximum of 16,5 hours a week.
2. A short code review highlighted the developers delivering low-quality, too complicated non-maintainable code in critical features. Most such developers are great engineers. Still, during their careers, they have learned a bad habit of developing at least one feature in a way that the feature could be maintained only by them, so the organization would need to pay a very high price if they decide to replace him and hire him as an outside consultant after to make the transition smooth. 7 developers out of the team of 52 are quite a high level. Considering their seniority level and payments on future options, the damage only on their salary and bonuses reached ~$4M/year. Damage dealt to the product itself is hard to calculate.
3. Interviews with core engineers and architects and code review showed critical parts of the system that would become scaling bottlenecks in a 12-month timeframe according to the development plan considering the traffic increase and poor code quality that leads to poor performance. It was not visible on the traffic client had at that moment, but the expansion plan would 20x it, and tech decisions made before would necessitate rewriting these bottlenecks on a ‘hot’ system. The position of these developers was quite high and they were not limited in time or have been put on a pressure, so it made clear that this was done on purpose.
4. Technical debt analysis during the interviews with product owners and sales team representatives showed that 2/3 of it would have demand from current prime clients product managers working with, and 3/4 of it would be good sales material for the sales & marketing team to promote the product and convert leads.

Solution

The plan development for the client has a few iterations:

1. Cleaning and preparations (0-2 months).
   • The most critical part at that moment was developers damaging the product. We could not just fire them because it would significantly slow down the development and would put its performance at risk. We have gradually put other developers who didn’t have this problem so that they can be injected into these critical features and infrastructure parts with current developers and get shared knowledge about it during the code review and refactoring process. After two and a half months, the organization could fire all seven developers damaging the company without damage to the product.
   • In parallel, we developed a plan of refactoring or redeveloping parts on the proper technology basis that were predicted to have scaling issues.
   • After cleaning up, the cloud part of the team would be the most naked, so we decided to create Cloud Task Force. At that moment, besides 2 people who were going to get fired, only one back-end developer had some knowledge about the cloud infrastructure. It was enough to keep it running without problems for at least a few months, so we were safe to go. It was not perfect, and even the easy deployment process was significantly slower, but it’s a good thing to find it early, not later.
2. Cloud Task Force (1,5-4 months) The cloud infrastructure was a bottleneck for the client for a long time and was a mess because of the high turnover in these positions, low motivation of employees to do things right, and lack of relevant technical experience in company management. We started searching for Team Lead on week 6 after we started the discovery phase. This is the most critical moment in creating every Task Force. After that, we got 1 Senior and 1 Middle cloud engineer there (junior developers are usually added in later stages when the Task Force reaches ~80-90% of possible perk productivity)
3. 2 more Task Forces (5-12 months)
   1. By this time, cloud Task Force showed incredible results. All infrastructure got to the shape it was intended to be. 40% of backlog completed during 4 months. The results impressed the client, and he decided to implement the same approach with us for 2 Task Forces. One product manager was taken from the client’s core team, and one was hired from outside; both trained for 3-4 weeks to understand the Task Force approach.
   2. While Product managers were approaching the Task Force method – 2 Team Leads were hired for these teams and started interviewing middle developers.
   3. Senior developer, then middle+ and middle-developers hired for each Task Force.

Results

1. 7 developers damaging the product were laid off. 2 project managers were laid due to performance issues (~10 month growth of technical debt growth per developer in their teams).
2. Cloud task Force caught up with all the cloud infrastructure and related tasks. Downtime, security, and overall cloud and deployment processes improved significantly.
3. 2 web development Task Forces were deployed on the project. Technical debt collected during 4 years of project life is reviewed, prioritized, and reducing 3-4% monthly. Sprints are now planned and estimated more accurately. 95% of tasks scheduled for the sprint are done on time.
4. The average time on calls per team member is reduced from 15h per week to 7h (which was highly appreciated by both delivery and management). The average cost per meeting is reduced from 960$/h to 375$/h.
5. 3 more task Forced planned to be deployed during the next 3 quarters.

Testimonials

“We choose Deventor for their passion for their approach to small teams. I felt it on the first call, but ever since, they proved to me that they fully understand how things are done. While everyone is trying to sell you ‘cost efficient’ cheap developers, the Deventor developers are doing an amazing job.”

Clutch