Reliability and Operations
Review configuration drift before changing application code
An unchanged application can behave differently after its operating settings change. Compare intended configuration with effective settings, classify approved exceptions, and test a narrow explanation before committing to an application rewrite.
When a feature breaks without an obvious release, changing application code may feel like progress. First ask whether the service still runs with the settings the team intended. Configuration drift means a difference between the approved baseline and the effective setup. The useful question is not whether two files differ, but whether a specific difference explains the observed business symptom.
Define what counts as the baseline
Identify the approved application version, operational settings, dependency references, and environment-specific exceptions. Give each exception an owner and a reason. Avoid treating the test environment as a perfect copy of production when intentional differences exist.
Separate secret references from ordinary settings. OWASP describes a secret lifecycle that includes creation, rotation, revocation, and expiration.[5] A changed secret dependency therefore deserves a lifecycle check, not an assumption that the application suddenly became defective. Compare authorized metadata without exposing the secret value.
Compare intended and effective behavior
Ask for a controlled comparison of the settings actually used by the affected service. A stored configuration file is only part of that question. Record the scope of the comparison and any places the team cannot observe. Keep the result limited to information reviewers are authorized to see.
Group differences into approved variation, unexplained change, and irrelevant detail. For each potentially relevant difference, write a hypothesis that predicts an observable effect. Do not restore every setting to an older state simply because that state is familiar.
Configuration review is not an excuse to delay every defect fix. If the effective setup matches the approved baseline, or a controlled test rejects the hypothesis, move the investigation forward. Preserve the negative result so the team does not repeat the same comparison.
Strong change controls also require effort and can slow legitimate emergency adjustments. Permit authorized exceptions, but record their purpose and review trigger. The outcome should be an explained operating state, not an unrealistic rule that every environment must always look identical.
Hypothetical example: a missing approval step
In this hypothetical example, an expense application stops requesting a second approval for one department. The application version matches the approved release. A review finds a changed department rule in the effective configuration.
The team reproduces the behavior in a controlled test and checks both the old and intended rule. The process owner confirms the correct policy before any authorized correction. This is an illustrative investigation, not evidence that configuration always explains unexplained failures.
Use a difference review checklist
Request a short review record before approving code changes whose purpose is to compensate for an unexplained setting.
- Name the approved baseline and the person who owns it.
- List intentional differences for the affected environment.
- Compare effective settings without revealing secret values.
- Connect each relevant difference to a testable symptom.
- Document the authorized correction and a way to reverse it.
