ApiumhubIn their thought-provoking presentation titled “Pragmatic Approach to Architecture Metrics” at GSAS’22 organized by Apiumhub, Sonya Natanzon, and Vlad Khononov delivered valuable insights. They offered the audience ample food for thought, emphasizing the importance of considering relevant metrics and adopting a pragmatic approach shed light on understanding the objectives of metrics and the need to make informed decisions about what to measure.
Pragmatic Approach to Architecture Metrics
The importance of architecture metrics
With metrics, we start thinking in terms of …
What gets measured, gets managed -Peter Drucker
But it may be more appropriate to rephrase it as …
What gets measured, gets managed — even when it’s pointless to measure and manage it, and even if it harms the purpose of the organization to do so
-Simon Caulkin on V.F. Ridgeway’s paper “Dysfunctional Consequences of Performance Measurements”, 1956
Not everything that we can measure necessarily holds significant importance, and conversely, there are crucial elements that defy easy measurement. The key lies in striking a balance between measurable indicators and a comprehensive understanding of the broader purpose and objectives of the organization
What do we want to measure?
We want to know if software architecture is successful
Everyone wants quantitative metrics to prove success
Software architecture is nebulous and difficult to measure
Ralph Johnson aptly states, “Architecture is about the important stuff. Whatever that is.”
The purpose of architecture lies in its alignment with the business goals, emphasizing that what truly matters for architecture is what effectively supports those goals
Identify what architectural qualities are important to support the business goals
See if you can measure them. These are your architecture metrics
(Successful) Product Evolution
The journey of software architecture involves Inception, Evolution, Market Adoption Expansion, and Scaling. It is crucial to recognize that Evolution and Scale represent distinct dimensions of change.
The success of software architecture hinges upon its adaptability to meet evolving business requirements. Consequently, we assess the capacity of architecture to embrace change through various metrics. It is important to acknowledge that the strategies that brought us to our current state may not suffice for future growth. Metrics should align with business progression rather than being bound by architectural considerations, necessitating their continual evolution.
But be very careful about metrics, because…
When a measure becomes a target, it ceases to be a good measure. – Goodhart’s Law
For example:
Code Coverage
Bugs found
Deployment frequency
Crash rate
Some good metrics to take into consideration:
Change Frequency
How often can we make meaningful changes?
Change Effectiveness
Did the change bring value to the users?
Change Cost
What does it take to make a change?
From Accelerate book
Complexity
As complexity escalates, so does the cost associated with change. The cost of change is intrinsically linked to the complexity of the modifications required. Consequently, our goal is to comprehend and evaluate the complexity of architecture. Complexity, as Geoffrey West aptly puts it, “You know it when you see it”.
Complexity in the Cynefin Model
• It is characterized by the relationship between cause and effect
• How predictable is the outcome of an action?
• In complex situations, the outcome can only be determined through empirical experimentation.
For example:
• What’s going to happen if I change this line of code?
• A change breaking unrelated parts of the system
• Not knowing what modules are affected by new business requirements
What Causes Complexity?
- Complexity arises from the interplay of system design and our cognitive capacities. It is influenced by the intricacies embedded within the system’s structure and behavior, as well as our ability to comprehend and navigate its complexities using our cognitive faculties.
• Estimated capacity for processing information in 1950s: 7 +/- 2
Miller, G. A. (1994). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 101(2), 343–352
• Estimated capacity for processing information in 2000s: 4 +/- 1
Cowan, Nelson. “The magical number 4 in short-term memory: A reconsideration of mental storage capacity.” Behavioral and brain sciences 24.1 (2001): 87-114.
Evaluating Complexity
Metrics play a crucial role in addressing complexity within systems, enabling us to quantify and assess the intricate aspects of system design and cognitive abilities.
- Abstractness Metric
The ratio of the number of abstract classes and interfaces in the analyzed package to the total number of classes in the analyzed package. (Wikipedia)
Abstractness (A) = number of abstract types/number of concrete types
- Instability Metric
The ratio of efferent coupling (Ce) to total coupling (Ce + Ca).
This metric is an indicator of the package’s resilience to change.
I=0 indicates a completely stable package
I=1 indicates a completely unstable package.
Instability (I) = no. of types you depend on / (no. of types you depend on + no. of types depend on you)
- Distance From the Main Sequence (D)
The perpendicular distance of a package from the idealized line A + I = 1. This metric is an indicator of the package’s balance between abstractness and stability. A package squarely on the main sequence is optimally balanced concerning its abstractness and stability. Ideal packages are either completely abstract and stable (I=0, A=1) or completely concrete and unstable (I=1, A=0).
D=|A+I-1|
COST OF CHANGE IS PROPORTIONAL TO THE DISTANCE BETWEEN COMPONENTS
The purpose
The goal of evaluating complexity is to identify design decisions that contribute to an increased cognitive load, ultimately leading to higher levels of complexity.
In this first part, we have explored the importance of metrics and evaluated complexity within a software architecture. However, there is much more to uncover. Stay tuned for the second part, where we will delve into metrics for success and the concept of integration strength. Join us as we continue this insightful journey towards a pragmatic approach to architecture metrics.
