The process of QA testing came into prominence, at least within more mainstream audiences, when stories about it came out regarding the popular (and some might say) infamous videogame Cyberpunk 2077, which has become known as one of the most high-profile disasters of shipped software products.
And as bugs were some of the most notorious problems of this game, it also presents the opportunity to talk about a very important part of software development, which can make or break a product: Quality Assurance. What does a good implementation of QA look like? What does it aim to find, and what are the best ways to go about it? And more importantly, what makes a good QA process?
1. Quantity of over quality (assurance)
An anonymous source within Quantic Labs, one of the firms in charge of QA in Cyberpunk 2077, told journalists about a “bug quota” imposed by management on their testers. With the requirement of reporting at least 10 bugs per day, the logic seemed to make sense: encourage your testers to be as thorough as possible, and thus ensure the final product will have the highest quality. However, if you are familiar with how the QA process works, you are already wincing because you know where this is going.
Quality Assurance is an important part of software development that, by ensuring that code is well-tested and meets standards, helps to improve the efficiency of a development team, and several good practices can help to ensure a successful quality process. Quotas can have the precise opposite effect, slowing down development by flooding developers with meaningless reports. Which, should be said, is no fault of the tester; after all, what is the result if they fail to meet them?
“I’ve worked in QA since 2011, been a Team Lead for the last three years, and bug quotas are a bad system which achieves absolutely nothing good”, explains one of the comments in the aforementioned article.“First, your testers enter a load of unproductive bugs, because they will divide one issue up into as many possible JIRA entries as they possibly can. And on top of that, they don’t have time to properly investigate the more complicated issues that they find — you get a lot more crash bugs with horribly elaborate reproduction steps because testers can’t afford to spend two hours nailing down exactly what triggers them.”
So with measures like these, the management of a project can unintendedly encourage bad QA, as it makes it more about the number of bugs found and less about their importance. So instead of quotas, a better method could be to ensure that everyone knows the priorities of the project, and have a clear definition of what constitutes a bug or issue (to let developers know when they need to fix something). Understand that QA is an important part of the development cycle from beginning to end, and enough time to do proper research and testing on bugs and issues is crucial in the planning of any successful project. Speaking of which…
2. Not a step, but an ongoing process
One of the biggest myths about QA testing is that it’s a one-time event that happens at the end of development. This simply isn’t true. Even if many people think it’s just a matter of catching bugs before a product is released, QA testing is an essential part of the software development process, and it should be treated as an ongoing collaboration between developers and testers.
This means regularly testing code and providing feedback to developers throughout the software development lifecycle; after all, effective QA testing is about making sure that a software application meets the requirements of the end-user. This means ensuring that the app is easy to use, bug-free, and performs well under real-world conditions, and QA testers play a vital role in ensuring that these standards are reached by working closely with developers throughout the whole project.
“Companies that realize the importance of Quality Assurance encourage employees to look at every part of the software development process as a “product” that has its consumer”, is a good explanation given by this blog from the QA firm Syndicode. “Defects are possible at each stage, so it’s important to ensure all participants adhere to the quality standards.”
After all, QA testing needs to be a collaboration between developers and testers, but it can also be heavy on time and resources. One way to improve efficiency and reduce costs is to ensure that a team, by working together, can quickly identify and fix errors, saving time and money in the long run.
3. Good communication between the QA team and developers is everything.
In any line of work, good communication is essential to collaboration, and this is especially true in the field of QA Testing, where clear and concise communication can mean the difference between a successful project and a costly mistake.
This means setting expectations, outlining the scope of the project, and establishing a clear process for reporting bugs and feedback. Without good communication, it can be difficult to get everyone on the same page, leading to frustration and delays. By taking the time to establish good communication early on, you can save yourself a lot of headaches down the road.
Also, this is where the advantages of a different approach in collaboration can shine, like the option of working with a Nearshore organization to find the QA talent your project needs. There are many benefits to this, like the increased diversity you get when expanding your scope (important to get as many fresh perspectives as you can when solving a particularly thorny bug), as well as efficiency and communication. With Nearshore proximity, it gets easier to build strong working relationships with other team members, whose cultural closeness makes work smoother in general, while also being flexible and scalable, making it a good option for businesses of all sizes. This way, teams can work more closely together to identify and resolve issues more quickly.
The result is that, with a Nearshore QA department, collaborative testing can also help to improve communication and build trust between team members; when everyone is on the same page, it leads to better quality software and a better user experience.
QA: More than meets the eye
The case of Cyberpunk 2077 we mentioned at the beginning is a great example of a QA process done wrong, and thankfully, any future product development can learn from it and understand how to approach an area of IT that sometimes doesn’t seem as valued as it should. The main thing is that proper QA is critical for success, and having a good approach towards it is the first step to guaranteeing a useful product that meets the expectations, and preferences, of a user base.
The Key Takeaways
QA is a critical part of software development, and any successful product has a strong quality process in place.
However, it’s very easy to choose the wrong approach to QA, compromising the functionality and success of any application, no matter how good it is.
Collaboration, communication, and a proper system that encourages looking for big issues during development are crucial, keeping everyone on the same page and with the same goals.
And when it comes to remote collaboration, a Nearshore partner is the best choice to bring the best QA talent to your team, as the close cultural fit and ability to communicate are invaluable to ensure a quality application.
Scio is an established Nearshore software development company based in Mexico that specializes in providing high-quality, cost-effective technologies to help you reach new heights. We have been developing since 2003 and our experience gives us access not only to the knowledge but also the expertise needed when tackling any project. Get started today by contacting us about your project needs – we’ll be happy to help you achieve your business goals.
with contributions by Ivan Guerrero Cardoso & Víctor Ariel Rodríguez Cruz.
It may seem like a pretty simple question, but how will your career look five years from now? What is your current goal? Because no matter if you are a junior, senior, or lead developer, the landscape of career progression is looking less like a straight path, and more like an open field of possibilities in which unexpected talents and skills can flourish.
So even if you are just starting out and trying to figure out your best course of action, or are a veteran whose path was full of unexpected twists and turns, we would like to ask you: what’s the shape you want your career in to take?
The Peter Principle
Now, before we start, here’s some context on why career paths have been getting more and more fluid, with skill sets becoming more diverse than ever during the last decade, and a good starting point is the infamous “Peter Principle” described by the Canadian educator Laurence J. Peter, which analyzes an issue that companies and organizations have been struggling with since times immemorial.
To put it in simple terms, the Peter Principle states that “a person that keeps getting promoted will eventually end up in a position beyond their competence, which will prevent them from further promotions, and thus keeping them in positions they fail at”, meaning that we all have a “competence ceiling” in our talents that we’ll probably hit sooner or later, depending on the career progression chosen. In more traditional companies this was a common issue when promoting people to a leadership position, like Management, where the best worker in a given department (let’s say, Sales) will be the first in line to get promoted when the opportunity presents, even if their best skill (selling) has nothing to do with the responsibilities of the promotion (in this case, managing people).
However, since this was the obvious career path for such a job, and rarely there was an alternative within the same organization, this meant that a person would have to “climb up” even if they had to stop doing the thing they were good at. Sometimes it worked out, sometimes it didn’t, and you can still see it today in programming and software development, where your best engineer, coder, QA, or tester excels because they are good and passionate at what they do, but, unless they explicitly seek to become leaders or managers, there is no guarantee of good performance in a new role with very different responsibilities.
Good companies, though, recognize this pitfall. “Careers are more organic today. Many organizations need a greater breadth of expertise and are encouraging employees to branch out, experiment with different functions, and become generalists rather than specialists. All of this opens new – and more – opportunities for people to grow, become engaged, and thrive at work”, says Skip Richard about the new realities of careers in the blog “Promotions are so yesterday”, which analyzes how career development is getting away from these traditional models, towards a more flexible approach. But what does this flexibility mean, and what does it look like in action?
The new realities of a career
What a lot of modern workplaces are trying to do nowadays is to get away from a linear concept of progress, offering their collaborators the chance to explore and develop new skills along with the flexibility to put them to use during a project. What if, for example, a senior Full Stack developer is interested in branching out to QA? Or if a member of the IT department wants to get into programming? Or what if a member of the marketing team is interested in learning to code?
“Career development used to be linear. It was all about that upward progression. Lateral – or God forbid downward – moves frequently reflected poorly upon ambitious professionals. […] Now, career development is much squigglier. People can move up, down and all around. They can go away and return – not something that was typical in the past”, continues the aforementioned blog.
And we are sure you have a pretty good guess on why this is the case; our relationship with work (and thus, careers) is changing, and following a single, inflexible path “forward” is less practical and realistic in a world that requires so much more from us, and the smartest organizations are those realizing that offering opportunities to explore new paths benefits us all.
“Until recently, I’d heard stories on the internet about software developers with peculiar paths, but never knew anyone like that. But then, one of my friends from high school, who studied Automation Engineering, started working as a software developer. Our CEO [Luis Aburto] has a background in Environmental Engineering, and some of the developers giving classes on the Apprenticeship told us about a couple of developers here with non-traditional paths, coming from Accounting or Chemistry. All these persons have inspired me to keep pursuing the career change I want”,says Iván Guerrero Cardoso, a Pharmaceutical Chemist who is currently an Application Developer Apprentice, a career he was able to pursue at Scio after discovering (and falling in love with) software development.
Stories like these are not uncommon at Scio and the software industry at large, and we will start to see how the ability to change paths not only will result in stronger organizations but in people less likely to get trapped in the Peter Principle, giving alternatives that let them explore their talents thoroughly.
“About a year and a half ago, I didn’t know anything related to software development as a profession — although I was already programming since a few years ago — and I found in Scio a place to learn, practice, and develop my skills even further. I’m also a follower of different web development personalities who has inspired me to try new things; I’d love to explore Web Development fully, Cyber Security is a topic I want to start leaning towards, and Videogame Development is an area where I’m constantly in the loop of the newest technologies”, says Víctor Ariel Rodríguez Cruz, a full-stack Application Developer at Scio, which touches on why this diversification is so attractive, beyond pure technical prowess: it lets us be part of a bigger whole.
The many dimensions of a career
“Humans have a deep need to connect, build relationships and be part of a community. Approached with intention, this can drive powerful career growth”, says Richards, and in the case of Scio, this idea of human connection [LINK] drives a lot of what we try to accomplish as an organization.
The reason is simple: in today’s world, where the ways we approach work are becoming more diverse than ever, Scio believes that investing in the personal growth of our collaborators, offering, among other things, paid technical courses, technical certifications, English classes, and our very own Leadership, Apprenticeship, and Sensei-Creati Coaching & Mentoring Programs to develop the hard and soft skills necessary to ensure the best collaboration between every developer, is the best way for a Nearshore Software Development company to foster the very best talent in Latin America.
This is a core part of the ways Scioneers can build their career prospects, encouraged by a culture of mutual support were learning a new skill can go from full-on coaching with someone with a vast experience to draw from, to simply asking for tips and pointers at informal meetings.
“Let me highlight the development dimension of ‘connection.’ This is one of my favorites, in large part because it’s become increasingly important in today’s distributed workplace”, continues the article by Richards. And this connection is built by also offering ways for our collaborators to develop the soft skills necessary to work on, collaborate or lead a team. Coming back to our example of the salesman becoming a manager, would it make a difference if such a promotion came with the coaching and tools necessary to learn how to handle a leadership position? The best way to give feedback, offer criticism, correct the trajectory of a project and balance the wants and needs of a team is not innate to many people, but that doesn’t mean it can’t be learned and developed.
Careers now need more than one dimension of growth, and Scio tries to offer the support to do just that: grow in the areas you want to.
Or as Ivan Guerrero puts it, about his own experience in the Scio apprentice program: “Not every company opens its doors to people without a strong computer science background, or without a related degree (which I understand, as it could be risky), therefore I’m hugely thankful with Scio for letting me join the team of apprentices where, as I see it, how passionate and willing we are to learn matters more than having an enormous amount of previous knowledge”.
The Key Takeaways:
Hitting the “Peter Principle” meant that a company didn’t offer enough options to choose from to advance a career, and that could lead to poor outcomes.
Today, a good company has multiple paths a collaborator can take, from traditional career advancement to coaching, to giving the opportunity to explore alternative options.
Career growth cannot happen in a vacuum either; a culture of collaboration can foster relationships that cross-pollinate knowledge and helps to open new paths to explore in a career.
Untangling how deeply our life changed during the Covid-19 pandemic will take a long time, and while we are still dealing with much of its aftermath, the process of planning the future doesn’t stop, even if we aren’t quite sure of what’s next for the industry.
“You can’t just link years of experience to the number of different technologies you may know. Being a Senior or Lead developer includes soft skills.”
— Helena Matamoros, Human Capital Manager at Scio.
When it comes to the difference between junior and senior developers, drawing a clear line of separation is more difficult than you would think. On one hand, a developer with two or three years of experience could still be learning the ropes of different roles, while on the other, a developer with between 5 and 10 years of experience keeps learning new frameworks and technologies every day. So if years of experience alone aren’t the most reliable metric, what concrete characteristics differentiate between a junior and a senior?
This is an important question because determining where you stand is the first step to knowing where to take your career and how to make it happen, even if you find that an answer is not as easy to find. Forbes magazine in this article, for example, begins by analyzing knowledge as a metric of seniority:
“Career growth in software engineering (and many other areas) is determined by the depth of knowledge and the breadth of expertise. A project can scale both horizontally, and vertically – which requires a different set of skills and experience in practical applications following both patterns”, notes the article.
However, defining what this “knowledge” means and entails goes beyond mere technical proficiency; how both kinds of developers handle a project, their approach and the outcomes they offer have as much weight as everything else: “In practice, a senior developer […] can assess the risks, ask the right follow-up questions, determine the right workflow, analyze the possible bottlenecks and deliver a stable solution. A junior developer can solve a limited set of problems with a smaller subset of alternative solutions.”
Just like playing tennis
This concept of knowledge reminded me of an interesting essay about tennis that can help illustrate this idea more easily. Called “Loser’s Game”, and written by investment consultant Chares Ellis, this text evaluates the difference between professional and amateur players, sheding a light on the psychology of a person trying to master a skill.
In short, a professional tennis player wins by earning more points than their opponent during a match (playing a “winner’s game”), which seems to be a sensible strategy for a competitive sport. However, it comes into focus when you learn that an amateur player has the exact opposite attitude: they try to win by losing fewer points than their opponents, playing a so-called “loser’s game”.
In other words, a professional tennis player tends to be more offensive, risking more for higher rewards. An amateur player is more defensive, setting on “safe” strategies that result in fewer risks and thus, lower rewards. And this mindset comes from both players having very different perspectives on the game: a professional plays long term, focused on the final result only, while an amateur focuses on each play, paying more attention to the short term gains.
Translating this to software development, we can say that a professional focuses more on outcomes, or the final result of the development process, while the amateur deals in outputs, or the result of each individual step of the project, and one of the key steps in the path of becoming a senior developer (or tennis player, for that matter) is recognizing and overcoming the difference between both approaches.
“Senior developers realize that software is very complex. It’s not possible to identify all the requirements upfront, they will change, there will be more requirements found, and know the project is not likely to hit the deadlines, or the software will be wrong on the early iterations. This isn’t because of bad development, it’s because of the creative process of creating software. Junior developers take “wrong” software personally, even if creating the wrong software is the path to creating the right software”, explains the blog ITNEXT in their article about this topic.
This, more than anything else, is where a line can be drawn indicating the difference between Junior and Senior developers; beyond just knowledge (which informs every individual approach), the way both types of developers conceptualize development, as well as how they collaborate with the rest of the team, is the threshold.
“This question is somewhat difficult to answer”, says Helena Matamoros, Scio’s Human Capital Manager, when presented with the question of how Scio defines the required seniority for different positions. “You can’t just link years of experience to the number of different technologies you may know. Being a Senior or Lead developer includes soft skills, such as leading teams, supervising the work of others, assigning tasks, reporting statuses, and visualizing obstacles in development, among other things. If you have been a developer for 10 years without these kinds of responsibilities, at least for Scio, you cannot be considered Senior.”
A quick start-up guide to seniority
Following this logic, then, we can begin to define a junior and a senior by the level of autonomy they can have during the development cycle of a product. This autonomy can include everything from technical knowledge (that is, how much supervision a given individual needs to complete a task), to their capacities to see the full picture of development (especially what concerns decision-making and direction of the project), so we can go back to the framework that the “Loser’s Game” essay defines for tennis players, as well as the characteristics defined by Helena, to start painting a broad picture of both types of developers:
Junior Developer:
Requires a higher level of support from their teammates.
Avoids making decisions without consulting superiors.
Weights risks in terms of outputs.
Prone to fixate more on the features of the product rather than its value as a whole.
Consider their mistakes a failure to avoid at all costs.
Less than five years of experience in a professional setting.
Senior Developer:
Requires a lower level of support, and may bring it to their teammates.
Is capable of analyzing the project and making their own decisions about its direction.
Weights risks in terms of outcomes.
Tends to consider more the value of the product as a whole.
Considers mistakes as a normal part of the process, and learns from them.
More than five years of experience in professional settings.
The last points in both definitions are included because even if the experience doesn’t dictate seniority, the skills necessary to act with autonomy, technical expertise, and leadership need time to develop, so engineers, programmers, and developers with a few years in their resumes are more likely to have the practice necessary to master this level.
This, however, is not a definitive list of characteristics, but an assessment of the behaviors that result from experience developing software, and it doesn’t even go into depth about the skill sets and knowledge that come from years of collaboration to bring the perfect outcome.
What we are trying to convey is that technical know-how is just the starting point; in collaborative environments where a team works with a client to achieve a specific goal, honing soft skills, getting comfortable with mistakes, bringing every learned lesson to the next project, and knowing how to manage expectations, resources, time and outputs is what a Junior developer needs to take the next step in their careers.
After all, growing as a developer is similar to growing as an adult; responsibilities get more important, you start paying attention to things you didn’t before, and before you know it, you are making your own way in the world. So what’s the next step you want to take?
When we think about the landscape of software development, it’s easy to fall into a binary view: people either work at a tech start-up, with the allure of innovation and cutting edge technology, or they work with a corporation like Google, Microsoft, or Amazon that promises stability and long term rewards in exchange for creating products with a more incremental and risk-averse pace.
There’s plenty of content about what it is like to work in both kinds of organizations, with lots of writing devoted to discussing the virtues and drawbacks of each, but are those the only two options in software development? What happens with mid-sized software companies, and in which ways do they distinguish themselves from either start-ups or big companies?
Outputs and outcomes
Let’s try and define the actual differences between these three types of companies, specifically in regards to how they view and measure product development objectives. After all, a software organization can be defined by the outcomes they want to achieve, and the outputs they decide to focus on to reach them. But what do these terms mean? In the words of Kirstie Magowan, from the BMC Blog:
The outcomes are what the business wants or need to achieve.
The outputs are the actions or items that contribute to achieving it.
Or in other words, “outcomes are the results, and outputs are the activities that support the desired results”, and clarity of purpose for both of them defines what a given organization focuses on to develop the best possible products.
So, within product development for software companies, outcomes are things like new version releases, new features implemented for the users, faster development cycles, improved quality, etc. On the other hand, outputs are things like software requirements/user story documentation, UI/UX prototypes, code, database scripts, test cases, DevOps scripts, and so on.
However, some nuance needs to come into play at this point since we can imagine that organizations of different sizes and maturity cannot all seek the same outcomes. We’ll explore it in more detail, but in our experience, the main force that drives start-ups, mid-sized companies, and big corporations is that their target user bases have different expectations of their respective products.
The weight of expectations in development
“The top desired outcome for a start-up development team, working on the early stages of a new product, is to build an MVP as fast as possible so the company can launch it and attract enough early adopters to keep iterating an idea”, says Luis Aburto, CEO, and Co-Founder of Scio, about his experience working with companies of all sizes. “At this point, you have a ‘forgiving’ attitude among the user base if the product shows enough potential. Early adopters will frequently overlook some issues if the value they get in return is enough and the product has a clear roadmap of features and functionalities that will improve the final version in the future.”
So the challenge for a start-up is to iterate a product fast enough to establish itself before interest dries up. After all, if we define a start-up as “a temporary organization designed to look for a business model that is repeatable and scalable”, the push for innovation is always running against time, looking to please early adopters enough so the product gets accepted in the mainstream. That way, you get organizations more willing to bend the rules, avoid the bureaucracy that comes with very defined processes, and overlook some production requirements (like documentation) if the team is small and their output is focused on delivering a product.
“In contrast,for a big company”, Luis continues, “every time you implement a new feature, you have an opportunity to either delight your users or cause a catastrophe. Suppose you are an organization that has maintained a product for 10 or 15 years. In that case, there are a lot of dependencies and expectations that come along with that, and if you are not careful, test everything closely, and research your market fully, you can end up in a position where you can damage your clients and users. What if they depended on certain features or functionality for their jobs, or had important data stored inside the product? The outcomes here would look very different, as the “forgivingness” of your user base gets lower the more mainstream it is.”
These expectations, then, dictate the best outcome for a given organization, and in turn, these outcomes will dictate the outputs needed to achieve the goal. However, the unique challenge here is that, while outputs can be defined and measured easily, the outcomes seek to “change a behavior in the user”, which is a much more subjective and nebulous goal. To this end, Natalie Diggings, of OpenView Partners, proposes a set of questions to evaluate the performance of any given team with an outcome-focused mindset that can help engineering teams and management alike to keep their goals aligned:
Are you delivering value for the customer?
Are you delivering results each quarter that will help grow the business?
Are you mitigating risk?
Do you know the ROI of the features you’re building?
Did you ship when you said you would ship?
“What we try to do at Scio, by implementing the Agile methodologies in every project we work on, is to establish a minimum of processes and continuous communication to ensure the output our team contributes to the desired outcome for the product, avoiding misaligned expectations and other issues”, continues Luis about Scio’s experiences working with start-ups. “And for a big company, we try and secure a seamless on-boarding of our developers, as these organizations have clear and defined processes that we like to follow to a T to fulfill these same expectations.”
Mid-sized companies: Hitting the sweet spot.
However, Scio’s area of expertise is a segment of the software development industry that doesn’t seem to have that much attention directed at the mid-sized range of companies that hit the balance between a start-up and a big corporation, which has some unique benefits worth exploring.
For example, if we go back to expectations, outputs, and outcomes, an organization of around 50 to 100 people is still guided and directed by the vision of its founders, which seems to bring some advantages that balance out the issues of the other two types of companies.
“Working with a big, Microsoft-like corporation seems like a no-brainer, but at that size, and the bureaucracy that comes along with it, there’s a very limited impact you can have in a given project. But with a medium-sized organization, where you still have the opportunity to work directly with the founders and share and understand their vision, alongside an organization still looking towards the same clear goals, collaboration becomes a more personal matter. Is a business with a human element still felt”, Luis concluded.
These kinds of connections are important when you look for a partner to help you reach a specific outcome, as a “clarity of purpose” can still be shared among the development teams, executives, and product owners, with the benefit of the structure, processes, and methods of a mature organization. After all, before hitting the “millions of users” metric, a mid-sized company still has the chance to push innovation forward with minimal risk.
This is also another big difference between these three kinds of organizations; for a start-up, risk in pursuit of innovation is the name of the game, and for a big corporation, incremental improvement with stable growth is the norm. Having an approach that can still make use of both perspectives is what makes working with these companies so interesting.
“On the business side, a mid-sized organization still looks to grow, but the relationship with their users and clients still has a personal touch, with support and communication at a human level, which allows them to get new clients more easily while still maintaining their current ones. And in the development size, the cohesion between collaborators, both in-house and outsourced, still has a clarity of purpose in their outputs.This is the way Scio functions, and also why we like to offer their support to mid-sized companies; our communication, collaboration, and goals always come hand in hand during a project. With this, the best outcomes are more measurable, reachable, and all in all, understandable”, concludes Luis.
The Key Takeaways:
An outcome is defined by the expectations of the clients of a given company, which should be taken into account in every development cycle.
Clarity of purpose is critical; if your dev team doesn’t know where their work is pointed towards, then it’s difficult for the output to match the outcome, which can vary heavily depending on the age and size of the organization.
Companies of different sizes have different strengths and weaknesses, but a mid-size company can have the best of both worlds: the processes necessary for a project to succeed, while still maintaining cohesion among its collaborators.
The personal relationships that can grow during collaboration are important for the outcome of a project, as they make it easier for a team of collaborators to share a vision and purpose in their outputs.
How to measure productivity effectively? Adolfo Cruz, Scio’s very own Project Management Office director, offers insights into a question many in the business have in their minds all the time.
By Adolfo Cruz.
Building an exact image of productivity is a question with no definitive answer yet. Defining what “productivity” means for someone starting a business for the first time is a complex process. From everything I’ve learned these 14 years at Scio, each person needs to discover what works for them, considering the specific context in which they are trying to succeed.
The thing is, when trying to measure productivity in something as fluid as software development, every seemingly useful metric comes from a different perspective (be it from the client, the developer, or the managers), so if you want to establish a baseline, I would recommend defining priorities first, taking into account your business needs.
How do we measure productivity in Scio?
Early in Scio’s history, around 2008 or so, we tried to implement Agile Methodologies along with some number crunching, like the hours put on a project or the number of bugs, to know exactly how productive our teams were. However, the number of variables involved, like realizing that a specific team had better outcomes because they had certain chemistry between them, made it difficult to translate the results into measures that could be implemented in any project.
So we moved away from that, instead of looking at the results from the client’s perspective, thus defining our productivity as the value our developers are adding to a product during development. Our measures became more focused on people than metrics because helping a client to build a product, adding value at every step, and therefore growing their business, was our main priority.
To this end, building a perfectly calibrated development team that proposes new things, learning and evolving together was the best approach for Scio. A lot of the time it’s not even necessary to keep a register of everything happening during development because the team itself realizes when something’s not right and works out any issue; a team of developers capable of self-management is always a good sign.
If you achieve such a team, you don’t need to measure specific metrics for each collaborator, instead, you evaluate the team as a whole, looking for relevant information that can help them grow, so you should think of questions that help you get the information you need.
For example number, many User Stories were completed in a given time?
Was there any serious bug in the development of the product?
How many bugs surfaced?
What priority did these bugs have?
How did the team respond to them?
How long did it take?
However, these questions need to be constructed with the project’s context in mind; although there are a lot of metrics you can use to generate specific “productivity” numbers (like the Personal Software Process popular in the 90s that tried to define productivity through the number of lines of code written), if they don’t reflect reality, then the team will not know how to use this information, as many of the challenges they face can be subjective.
So avoid trying to arrive at exact numbers, and instead, I recommend you to look at the trends emerging from one project to another. A trend interpreted correctly should give you a “big picture” of both performance and productivity, showing places where your attention or some improvement is needed.
First, look for a certain range of information in a development cycle; If the team completed between 5 and 10 stories in a sprint, but only 1 or 2 the next one, there’s probably a red flag somewhere in need of attention. Then dig a little into it, and maybe you’ll find some unforeseen challenges the team needed to prioritize to keep making progress. And finally, make a note out of it, so you can improve the circumstances around an issue so it doesn’t reappear later; that’s how an understanding of productivity begins to emerge.
Most of the trends we watch at Scio are about how our developers contribute to our client’s goals, and we developed some tools to measure that. For example, we use the Team Self-Assessment, a guideline for collaborators to reflect on their work, that we adopted and modified according to what our experiences and specific contexts dictate.
In our TSA, there’s a section where we inquire about bugs surfacing during development, asking how they are preventing issues during development, if they are doing Pair Testing practices, if they are showing their additions to someone else before moving on, and generally if they are focusing on things that matter.
When a team reviews these questions, they realize their areas of improvement, and we work together to avoid falling into them again in future projects, because what you want to do is push your teams into adopting excellence as a natural part of their processes, encouraging them to add their experiences and expertise to the final product.
The subjectivity of a productive team
Productivity is not always a purely technical approach; sometimes you need to look at the business side of things, involving the stakeholders and product owners in the creation of the product to reconcile a multitude of perspectives, converting them into priorities to define the best course of action.
Just remember that you need to experiment in software development, build an iteration of the product to validate its potential, and look for features that can be improved to maximize its success. Your development team will probably need to go back and rework a lot of stuff, updating their methods to create a more efficient version of the product, so your definition of productivity here has to account for it; if stuff is being redone, at least quality should always be improving.
So, to recap, a good measure of productivity is established by defining your business priorities and then choosing the metrics that best reflect the context in which the product will be built. A productive development process gives enough flexibility to iterate and experiment without neglecting business needs, and having your client’s goal as the focus of your team is a great starting point.
Just remember that if you aren’t adapting what you learn into your specific context, capturing the information your team needs, then productivity might suffer. Software is mostly a creative enterprise, so some subjectivity will always be needed to properly define what makes a successful project, fine-tuning every cycle to achieve the value you want for a specific product. With an approach like that, you can be sure your organization will always be improving.
How to measure productivity? That’s a question that many in the business, from CEOs to coders to engineers to managers, have in their minds all the time, and Adolfo Cruz, Scio’s very own Project Management Office director discusses metrics, measures, and meanings of productivity.
At the end of the 90s, a methodology called “Personal Software Process”, or PSP, was designed to help developers measure their productivity. You had to take a course, there was a lot of documentation to follow through, and you had to use a timer to measure your progress, stopping it every time you needed a cup of coffee or to go to the bathroom.
The idea was to see how much you accomplished in a day, but in fact, this methodology was entangled too closely with the number of lines you wrote, meaning that you were more productive the more you coded, which is not necessarily true.
But if this is not productivity, what is it?
I define “productivity” as finishing a task in a reasonable time. The word “finishing” here is key because productivity is not starting a lot of things, but seeing a project to completion, right until you get a product. However, how do you define exactly when something is “finished” in software development? What criteria do you have to fulfill to say something is done? If we were building something physical, let’s say a boat, first, you need to build a hull, and this phase ends when it fulfills certain requirements.
And although not all boats are the same (building a freighter or a yacht would look very different), in essence, you have the same process, blueprints, and requirements to do it. Software development doesn’t work that way.
Developing software involves a lot of things. You may see it conceptualized as a “factory”, or a development team working like a well-oiled machine, where you input requirements and get a product at the other end. But in truth, there’s an artisanal quality to developing software; everyone has their approach and style, and progress changes according to the team you are with.
This results in a lively debate where no one agrees on the best way to measure productivity. If you ask a room full of developers how many lines of code they write to see if they are productive, you can get a very heated discussion, because how are you measuring that?
The best code is concise and everyone checking it can understand it, so I don’t know how you can see someone writing 10,000 lines of code and conclude he is more productive than someone achieving the same in 500. Maybe it made more sense at a time with no frameworks to build things faster, when everything was a bit more rudimentary and you coded every single piece of the product, but today you have to write very few things from scratch, with a whole range of tools that let you produce, let’s say, the shell of a website in a minute without coding anything directly.
So imagine if a company starts giving productivity bonuses based on lines of code produced per project. They would end up with developers gaming the system to get the bonus or at least trying to not look worse than their peers by writing less, resulting in bloated, inefficient products because the incentive wasn’t centered on creating something useful.
You have to be very careful when linking rewards to metrics, or else you’ll generate a perverse environment where everybody is just racing to inflate numbers.
The Scio way
I’ve been with Scio for 14 years, and since then, perspectives have changed. With the arrival of Agile Methodologies, we moved on from counting lines of code to seeing how that code comes together, achieving working software whose process of development is not focused on numbers, but on how the product is going to be used.
To give you an idea of this evolution, not long ago the requirements of a project were written from the point of view of the system, so every requirement started with the phrase “The system shall…”: the system shall do X thing, the system shall do Y thing, the system shall do Z thing, etc.
So you ended up with a massive document repeating “The system shall…” for every little thing. Then the Agile Movement centered on the user, with requirements stating “The Administrator can…” or “The Manager can…” because we understood that software is not about a system that “shall” do things, but what people in different roles “can” achieve with the product, resulting in productivity built around how much value we give to the final user, not how much code our devs write.
Coming back to Scio, we see it from the perspective of the stakeholders and clients we are building a product for, and our productivity is measured on the information we get from them, knowing how our teams are doing, how much value they are adding to a project, and what their perception of the whole process is. It’s a more people-oriented approach, far from the days of counting lines of code, and more interested in understanding how a developer is contributing to the goals of our clients.
To that end, we developed some internal tools, like the Team Self-Assessment, based on our prior experiences, consisting of questionnaires about the things we consider important for a team to focus on. For example, there’s an entire section about quality, how they are preventing issues during development, if they are doing Pair Testing practices, or if they are doing code reviews to make sure the code is maintainable and scalable…
Are they giving issues the proper attention? Are they documenting them? The team members have to ask themselves if they are focusing on the right issues, and if they aren’t, it’s something we have to improve. That’s how we try to motivate our teams to share their experiences, practices, and insights into our client’s projects.
It is said that only 35% of software development projects succeed, and I think it has to do with the planning stage of a product. Let’s say I want to complete the A, B, and C steps of a project in six months, based on previous experiences in similar projects. But it ended up taking 8 months instead of 6 because something needed to change, does that 2-month delay mean the project is going wrong?
It happens a lot with start-ups trying to create something completely new. In the course of development, it’s common to see something, a feature or function of the product that changes the client’s perspective, that taps into some potential we haven’t seen before, so the plan has to get reworked to harness that and bring its full potential. In six months, priorities can change.
But if we measure the productivity of the process very rigidly, and then that very same process brings out the value in unexpected places that, nonetheless, force you to rework entire parts of the project, it’s easy to see it as a failure.
The Project Management Institute uses these rigid measures a lot, asking for a specific basis, beginning, and end of every phase of a project, and if you don’t deliver them exactly as planned, then you get a mark against you. In an actual software development environment, that doesn’t happen, because the dynamics of a development cycle can change fast.
Software development works by evolution
The measures you have to use are subjective more often than not. Some industries require strictness, especially when manufacturing something, but in the world of software, and start-ups in specific, I don’t think it’s necessary to be like this to create a successful product.
This is why we back away a little from the technical aspects of a project and focus instead on the business side of things, having conversations with stakeholders and product owners to get them involved, reconciling all these points of view about what the business needs, and how development is.
We take a look at the features planned, check how many people ask for them, how critical they are for the business model to work, and decide how to proceed from there, adding real value by focusing on building those pieces first. Technical aspects are solved later, as you first see what the business needs, and then the technical team starts sketching solutions for the challenge.
Productivity is a question with no definitive answer yet.
Considering all this, forming an exact image of productivity is a question with no definitive answer yet. Every individual has to decide what works, but only in the specific context in which that person is acting, so no one has come up with a universal method to measure productivity in software development, as even the perspective from which you measure can change; seeing it from a client’s perspective is a world apart from a developer’s.
As you discover better routes during development that weren’t foreseen during the planning stage, or maybe because the technical aspect ended up being unfeasible for one reason or another, or the infrastructure cost is too high for your purposes, or any other number of reasons, a lot of what you may define at the beginning of the project will change.
You adapt, which is very different from building furniture or producing food, where it is clear what you are trying to accomplish over and over. But in software, where there’s no single solution for the same problem, it’s difficult to reach a consensus on what you need to do in detail.
However, you want to measure productivity, metrics evolve, and whatever method you use to decide how productive your team or your company is, I think the Agile Methodologies got it right, where it doesn’t matter the number of lines, or the documentation you have, or how pretty your database is, what matters to the client and the final user is having software that works.
