A place to cache linked articles (think custom and personal wayback machine)
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

index.md 55KB

3 years ago
  1. title: Feature Toggles (aka Feature Flags)
  2. url: https://martinfowler.com/articles/feature-toggles.html
  3. hash_url: 7c569b2880bd3ed844e039a322b9731d
  4. <p>"Feature Toggling" is a set of patterns which can help a team to deliver new
  5. functionality to users rapidly but safely. In this article on Feature Toggling we'll
  6. start off with a short story showing some typical scenarios where Feature Toggles are
  7. helpful. Then we'll dig into the details, covering specific patterns and practices
  8. which will help a team succeed with Feature Toggles.</p>
  9. <p>Feature Toggles are also refered to as Feature Flags, Feature Bits, or Feature Flippers.
  10. These are all synonyms for the same set of techniques. Throughout this article I'll use
  11. feature toggles and feature flags interchangebly.</p>
  12. <div id="ATogglingTale"><hr class="topSection"/>
  13. <h2>A Toggling Tale</h2>
  14. <p>Picture the scene. You're on one of several teams working on a sophisticated town
  15. planning simulation game. Your team is responsible for the core simulation engine.
  16. You have been tasked with increasing the efficiency of the Spline Reticulation
  17. algorithm. You know this will require a fairly large overhaul of the implementation
  18. which will take several weeks. Meanwhile other members of your team will need to
  19. continue some ongoing work on related areas of the codebase. </p>
  20. <p>You want to avoid branching for this work if at all possible, based on previous
  21. painful experiences of merging long-lived branches in the past. Instead, you decide
  22. that the entire team will continue to work on trunk, but the developers working on
  23. the Spline Reticulation improvements will use a Feature Toggle to prevent their work
  24. from impacting the rest of the team or destabilizing the codebase.</p>
  25. <div id="TheBirthOfAFeatureFlag">
  26. <h3>The birth of a Feature Flag</h3>
  27. <p>Here's the first change introduced by the pair working on the algorithm:</p>
  28. <p class="code-label">before
  29. </p>
  30. <pre class="code"> function reticulateSplines(){
  31. // current implementation lives here
  32. }</pre>
  33. <p class="code-label">after
  34. </p>
  35. <pre class="code"> function reticulateSplines(){
  36. var useNewAlgorithm = false;
  38. if( useNewAlgorithm ){
  39. return enhancedSplineReticulation();
  40. }else{
  41. return oldFashionedSplineReticulation();
  42. }
  43. }
  44. function oldFashionedSplineReticulation(){
  45. // current implementation lives here
  46. }
  47. function enhancedSplineReticulation(){
  48. // TODO: implement better SR algorithm
  49. }</pre>
  50. <p>The pair have moved the current algorithm implementation into an
  51. <code>oldFashionedSplineReticulation</code> function, and turned
  52. <code>reticulateSplines</code> into a <b>Toggle Point</b>. Now if someone is
  53. working on the new algorithm they can enable the "use new Algorithm"
  54. <b>Feature</b> by uncommenting the <code>useNewAlgorithm = true</code>
  55. line.</p>
  56. </div>
  57. <div id="MakingAFlagDynamic">
  58. <h3>Making a flag dynamic</h3>
  59. <p>A few hours pass and the pair are ready to run their new algorithm through some
  60. of the simulation engine's integration tests. They also want to exercise the old
  61. algorithm in the same integration test run. They'll need to be able to enable or
  62. disable the Feature dynamically, which means it's time to move on from the clunky
  63. mechanism of commenting or uncommenting that <code>useNewAlgorithm = true</code>
  64. line:</p>
  65. <pre class="code">function reticulateSplines(){
  66. if( featureIsEnabled("use-new-SR-algorithm") ){
  67. return enhancedSplineReticulation();
  68. }else{
  69. return oldFashionedSplineReticulation();
  70. }
  71. }
  72. </pre>
  73. <p>We've now introduced a <code>featureIsEnabled</code> function, a <b>Toggle
  74. Router</b> which can be used to dynamically control which codepath is live.
  75. There are many ways to implement a Toggle Router, varying from a simple in-memory
  76. store to a highly sophisticated distributed system with a fancy UI. For now we'll
  77. start with a very simple system:</p>
  78. <pre class="code">function createToggleRouter(featureConfig){
  79. return {
  80. setFeature(featureName,isEnabled){
  81. featureConfig[featureName] = isEnabled;
  82. },
  83. featureIsEnabled(featureName){
  84. return featureConfig[featureName];
  85. }
  86. };
  87. }
  88. </pre>
  89. <p>We can create a new toggle router based on some default configuration - perhaps
  90. read in from a config file - but we can also dynamically toggle a feature on or
  91. off. This allows automated tests to verify both sides of a toggled feature:</p>
  92. <pre class="code">describe( 'spline reticulation', function(){
  93. let toggleRouter;
  94. let simulationEngine;
  95. beforeEach(function(){
  96. toggleRouter = createToggleRouter();
  97. simulationEngine = createSimulationEngine({toggleRouter:toggleRouter});
  98. });
  99. it('works correctly with old algorithm', function(){
  100. // Given
  101. toggleRouter.setFeature("use-new-SR-algorithm",false);
  102. // When
  103. const result = simulationEngine.doSomethingWhichInvolvesSplineReticulation();
  104. // Then
  105. verifySplineReticulation(result);
  106. });
  107. it('works correctly with new algorithm', function(){
  108. // Given
  109. toggleRouter.setFeature("use-new-SR-algorithm",true);
  110. // When
  111. const result = simulationEngine.doSomethingWhichInvolvesSplineReticulation();
  112. // Then
  113. verifySplineReticulation(result);
  114. });
  115. });
  116. </pre>
  117. </div>
  118. <div id="GettingReadyToRelease">
  119. <h3>Getting ready to release</h3>
  120. <p>More time passes and the team believe their new algorithm is feature-complete.
  121. To confirm this they have been modifying their higher-level automated tests so
  122. that they exercise the system both with the feature off and with it on. The team
  123. also wants to do some manual exploratory testing to ensure everything works as
  124. expected - Spline Reticulation is a critical part of the system's behavior, after
  125. all. </p>
  126. <p>To perform manual testing of a feature which hasn't yet been verified as ready
  127. for general use we need to be able to have the feature Off for our general user
  128. base in production but be able to turn it On for internal users. There are a lot
  129. of different approaches to achieve this goal:</p>
  130. <ul>
  131. <li>Have the Toggle Router make decisions based on a <b>Toggle Configuration</b>,
  132. and make that configuration environment-specific. Only turn the new feature on in a
  133. pre-production environment.</li>
  134. <li>Allow Toggle Configuration to be modified at runtime via some form of admin UI. Use
  135. that admin UI to turn the new feature on a test environment.</li>
  136. <li>Teach the Toggle Router how to make dynamic, per-request toggling decisions.
  137. These decisions take <b>Toggle Context</b> into account, for example by looking for a special cookie
  138. or HTTP header. Usually Toggle Context is used as a proxy for identifying the user making the request.</li>
  139. </ul>
  140. <p>(We'll be digging into these approaches in more detail later on, so don't worry if some
  141. of these concepts are new to you.)</p>
  142. <p class="clear"/>
  143. <p>The team decides to go with a per-request Toggle Router since it gives them a lot of
  144. flexibility. The team particularly appreciate that this will allow them to test their new algorithm without needing
  145. a separate testing environment. Instead they can simply turn the algorithm on in their
  146. production environment but only for internal users (as detected via a special cookie). The
  147. team can now turn that cookie on for themselves and verify that the new feature performs
  148. as expected.</p>
  149. </div>
  150. <div id="CanaryReleasing">
  151. <h3>Canary releasing</h3>
  152. <p>The new Spline Reticulation algorithm is looking good based on the exploratory
  153. testing done so far. However since it's such a critical part of the game's
  154. simulation engine there remains some reluctance to turn this feature on for all
  155. users. The team decide to use their Feature Flag infrastructure to perform a
  156. <a href="/bliki/CanaryRelease.html"><b>Canary Release</b></a>, only turning the new
  157. feature on for a small percentage of their total userbase - a "canary" cohort.
  158. </p>
  159. <p>The team enhance the Toggle Router by teaching it the concept of user cohorts -
  160. groups of users who consistently experience a feature as always being On or Off. A
  161. cohort of canary users is created via a random sampling of 1% of the user base -
  162. perhaps using a modulo of user ID. This canary cohort will consistently have the
  163. feature turned on, while the other 99% of the user base remain using the old
  164. algorithm. Key business metrics (user engagement, total revenue earned, etc) are
  165. monitored for both groups to gain confidence that the new algorithm does not
  166. negatively impact user behavior. Once the team are confident that the new feature has no
  167. ill effects they modify their Toggle Configuration to turn it on for the entire user
  168. base.</p>
  169. </div>
  170. <div id="AbTesting">
  171. <h3>A/B testing</h3>
  172. <p>The team's product manager learns about this approach and is quite excited. She
  173. suggests that the team use a similar mechanism to perform some A/B testing. There's been a
  174. long-running debate as to whether modifying the crime rate algorithm to take
  175. pollution levels into account would increase or decrease the game's playability.
  176. They now have the ability to settle the debate using data. They plan to roll out a
  177. cheap implementation which captures the essence of the idea, controlled with a
  178. Feature Flag. They will turn the feature on for a reasonably large cohort of
  179. users, then study how those users behave compared to a "control" cohort. This approach will allow
  180. the team to resolve contentious product debates based on data, rather than <a href="http://www.forbes.com/sites/derosetichy/2013/04/15/what-happens-when-a-hippo-runs-your-company/">HiPPOs</a>.</p>
  181. </div>
  182. <p>This brief scenario is intended to illustrate both the basic concept of Feature
  183. Toggling but also to highlight how many different applications this core capability
  184. can have. Now that we've seen some examples of those applications let's dig a little
  185. deeper. We'll explore different categories of toggles and see what makes them
  186. different. We'll cover how to write maintainable toggle code, and finally share
  187. practices to avoid some of pitfalls of a feature-toggled system.</p>
  188. </div>
  189. <div id="CategoriesOfToggles"><hr class="topSection"/>
  190. <h2>Categories of toggles</h2>
  191. <p>We've seen the fundamental facility provided by Feature Toggles - being able to
  192. ship alternative codepaths within one deployable unit and choose between them at
  193. runtime. The scenarios above also show that this facility can be used in various
  194. ways in various contexts. It can be tempting to lump all feature toggles into the
  195. same bucket, but this is a dangerous path. The design forces at play for different
  196. categories of toggles are quite different and managing them all in the same way can
  197. lead to pain down the road. </p>
  198. <p>Feature toggles can be categorized across two major dimensions: how long the
  199. feature toggle will live and how dynamic the toggling decision must be. There are
  200. other factors to consider - who will manage the feature toggle, for example - but I
  201. consider longevity and dynamism to be two big factors which can help guide how to manage
  202. toggles.</p>
  203. <p>Let's consider various categories of toggle through the lens of these two
  204. dimensions and see where they fit.</p>
  205. <div id="ReleaseToggles">
  206. <h3>Release Toggles</h3>
  207. <div class="soundbite">
  208. <p>
  209. Release Toggles allow incomplete and un-tested codepaths to be shipped to production as latent code which may never be turned on.
  210. </p>
  211. </div>
  212. <p>These are feature flags used to enable trunk-based development for teams practicing
  213. Continuous Delivery. They allow in-progress features to be checked into a shared
  214. integration branch (e.g. master or trunk) while still allowing that branch to be deployed to production at
  215. any time. Release Toggles allow incomplete and un-tested codepaths to be shipped to
  216. production as <a href="http://www.infoq.com/news/2009/08/enabling-lrm">latent code</a> which may
  217. never be turned on. </p>
  218. <p>Product Managers may also use a product-centric version of this same approach to
  219. prevent half-complete product features from being exposed to their end users. For
  220. example, the product manager of an ecommerce site might not want to let users see a
  221. new Estimated Shipping Date feature which only works for one of the site's shipping
  222. partners, preferring to wait until that feature has been implemented for all shipping
  223. partners. Product Managers may have other reasons for not wanting to expose features
  224. even if they are fully implemented and tested. Feature release might be being
  225. coordinated with a marketing campaign, for example. Using Release Toggles in this way
  226. is the most common way to implement the Continuous Delivery principle of "separating
  227. [feature] release from [code] deployment."</p>
  228. <div class="figure " id="chart-1.png"><img src="feature-toggles/chart-1.png"/>
  229. <p class="photoCaption"/>
  230. </div>
  231. <p class="clear"/>
  232. <p>Release Toggles are transitionary by nature. They should generally not stick around
  233. much longer than a week or two, although product-centric toggles may need to remain in
  234. place for a longer period. The toggling decision for a Release Toggle is
  235. typically very static. Every toggling decision for a given release version will be the
  236. same, and changing that toggling decision by rolling out a new release with a toggle
  237. configuration change is usually perfectly acceptable.</p>
  238. </div>
  239. <div id="ExperimentToggles">
  240. <h3>Experiment Toggles</h3>
  241. <p>Experiment Toggles are used to perform multivariate or A/B testing. Each user of
  242. the system is placed into a cohort and at runtime the Toggle Router will
  243. consistently send a given user down one codepath or the other, based upon which
  244. cohort they are in. By tracking the aggregate behavior of different cohorts we can
  245. compare the effect of different codepaths. This technique is commonly used
  246. to make data-driven optimizations to things such as the purchase flow of an
  247. ecommerce system, or the Call To Action wording on a button.</p>
  248. <div class="figure " id="chart-2.png"><img src="feature-toggles/chart-2.png"/>
  249. <p class="photoCaption"/>
  250. </div>
  251. <p class="clear"/>
  252. <p>An Experiment Toggle needs to remain in place with the same configuration long
  253. enough to generate statistically significant results. Depending on traffic patterns
  254. that might mean a lifetime of hours or weeks. Longer is unlikely to be useful, as
  255. other changes to the system risk invalidating the results of the experiment. By
  256. their nature Experiment Toggles are highly dynamic - each incoming request is likely
  257. on behalf of a different user and thus might be routed differently than the last.
  258. </p>
  259. </div>
  260. <div id="OpsToggles">
  261. <h3>Ops Toggles</h3>
  262. <p>These flags are used to control operational aspects of our system's behavior.
  263. We might introduce an Ops Toggle when rolling out a new feature which has unclear
  264. performance implications so that system operators can disable or degrade that
  265. feature quickly in production if needed. </p>
  266. <p>Most Ops Toggles will be relatively short-lived - once confidence is gained in
  267. the operational aspects of a new feature the flag should be retired. However it's
  268. not uncommon for systems to have a small number of long-lived "Kill Switches" which
  269. allow operators of production environments to gracefully degrade non-vital system
  270. functionality when the system is enduring unusually high load. For example, when
  271. we're under heavy load we might want to disable a Recommendations panel on our home
  272. page which is relatively expensive to generate. I consulted with an online retailer
  273. that maintained Ops Toggles which could intentionally disable many non-critical
  274. features in their website's main purchasing flow just prior to a high-demand product
  275. launch. These types of long-lived Ops Toggles could be seen as a manually-managed
  276. <a href="/bliki/CircuitBreaker.html">Circuit Breaker</a>.</p>
  277. <div class="figure " id="chart-3.png"><img src="feature-toggles/chart-3.png"/>
  278. <p class="photoCaption"/>
  279. </div>
  280. <p class="clear"/>
  281. <p>As already mentioned, many of these flags are only in place for a short while, but a few
  282. key controls may be left in place for operators almost indefinitely. Since the
  283. purpose of these flags is to allow operators to quickly react to production
  284. issues they need to be re-configured extremely quickly - needing to roll out a
  285. new release in order to flip an Ops Toggle is unlikely to make an Operations person happy.</p>
  286. </div>
  287. <div id="PermissioningToggles">
  288. <h3>Permissioning Toggles</h3>
  289. <div class="soundbite">
  290. <p>turning on new features for a set of internal users [is a] Champagne Brunch - an early opportunity to drink your own champagne</p>
  291. </div>
  292. <p>These flags are used to change the features or product experience that certain
  293. users receive. For example we may have a set of "premium" features which we only
  294. toggle on for our paying customers. Or perhaps we have a set of "alpha" features
  295. which are only available to internal users and another set of "beta" features which
  296. are only available to internal users plus beta users. I refer to this technique of
  297. turning on new features for a set of internal or beta users as a Champagne Brunch -
  298. an early opportunity to "<a href="http://www.cio.com/article/122351/Pegasystems_CIO_Tells_Colleagues_Drink_Your_Own_Champagne">drink your own
  299. champagne</a>".
  300. </p>
  301. <p>A Champagne Brunch is similar in many ways to a Canary Release. The distinction
  302. between the two is that a Canary Released feature is exposed to a randomly selected
  303. cohort of users while a Champagne Brunch feature is exposed to a specific set of
  304. users.</p>
  305. <p class="clear"/>
  306. <p>When used as a way to manage a feature which is only exposed to premium users a
  307. Permissioning Toggle may be very-long lived compared to other categories of Feature
  308. Toggles - at the scale of multiple years. Since permissions are user-specific the toggling
  309. decision for a Permissioning Toggle will always be per-request, making this a very dynamic toggle.</p>
  310. </div>
  311. <div id="ManagingDifferentCategoriesOfToggles">
  312. <h3>Managing different categories of toggles</h3>
  313. <p>Now that we have a toggle categorization scheme we can discuss how those two
  314. dimensions of dynamism and longevity affect how we work with feature flags of different
  315. categories.</p>
  316. <div id="StaticVsDynamicToggles">
  317. <h4>static vs dynamic toggles</h4>
  318. <p class="clear"/>
  319. <p>Toggles which are making runtime routing decisions necessarily need more
  320. sophisticated Toggle Routers, along with more complex configuration for those
  321. routers.</p>
  322. <p>For simple static routing decisions a toggle configuration can be a simple On
  323. or Off for each feature with a toggle router which is just responsible for relaying that
  324. static on/off state to the Toggle Point. As we discussed earlier, other
  325. categories of toggle are more dynamic and demand more sophisticated toggle
  326. routers. For example the router for an Experiment Toggle makes routing
  327. decisions dynamically for a given user, perhaps using some sort of consistent
  328. cohorting algorithm based on that user's id. Rather than reading a static toggle
  329. state from configuration this toggle router will instead need to read some sort of
  330. cohort configuration defining things like how large the experimental cohort and
  331. control cohort should be. That configuration would be used as an input into the
  332. cohorting algorithm. </p>
  333. <p>We'll dig into more detail on different ways to manage this toggle
  334. configuration later on.</p>
  335. </div>
  336. <div id="Long-livedTogglesVsTransientToggles">
  337. <h4>Long-lived toggles vs transient toggles</h4>
  338. <p class="clear"/>
  339. <p>We can also divide our toggle categories into those which are essentially
  340. transient in nature vs. those which are long-lived and may be in place for years.
  341. This distinction should have a strong influence on our approach to implementing
  342. a feature's Toggle Points. If
  343. we're adding a Release Toggle which will be removed in a few days time then we can
  344. probably get away with a Toggle Point which does a simple if/else check on a
  345. Toggle Router. This is what we did with our spline reticulation example
  346. earlier:</p>
  347. <pre class="code">function reticulateSplines(){
  348. if( featureIsEnabled("use-new-SR-algorithm") ){
  349. return enhancedSplineReticulation();
  350. }else{
  351. return oldFashionedSplineReticulation();
  352. }
  353. }
  354. </pre>
  355. <p>However if we're creating a new Permissioning Toggle with Toggle Points which
  356. we expect to stick around for a very long time then we certainly don't want to
  357. implement those Toggle Points by sprinkling if/else checks around
  358. indiscriminately. We'll need to use more maintainable implementation
  359. techniques.</p>
  360. </div>
  361. </div>
  362. </div>
  363. <div id="ImplementationTechniques"><hr class="topSection"/>
  364. <h2>Implementation Techniques</h2>
  365. <p>Feature Flags seem to beget rather messy Toggle Point code, and these Toggle
  366. Points also have a tendency to proliferate throughout a codebase. It's important to
  367. keep this tendency in check for any feature flags in your codebase, and critically
  368. important if the flag will be long-lived. There are a few implementation patterns
  369. and practices which help to reduce this issue.</p>
  370. <div id="De-couplingDecisionPointsFromDecisionLogic">
  371. <h3>De-coupling decision points from decision logic</h3>
  372. <p>One common mistake with Feature Toggles is to couple the place where a toggling
  373. decision is made (the Toggle Point) with the logic behind the decision (the Toggle
  374. Router). Let's look at an example. We're working on the next generation of our
  375. ecommerce system. One of our new features will allow a user to easily cancel an
  376. order by clicking a link inside their order confirmation email (aka invoice email). We're using
  377. feature flags to manage the rollout of all our next gen functionality. Our
  378. initial feature flagging implementation looks like this:</p>
  379. <p class="code-label">invoiceEmailer.js
  380. </p>
  381. <pre class="code"> const features = fetchFeatureTogglesFromSomewhere();
  382. function generateInvoiceEmail(){
  383. const baseEmail = buildEmailForInvoice(this.invoice);
  384. if( features.isEnabled("next-gen-ecomm") ){
  385. return addOrderCancellationContentToEmail(baseEmail);
  386. }else{
  387. return baseEmail;
  388. }
  389. }
  390. </pre>
  391. <p>While generating the invoice email our
  392. InvoiceEmailler checks to see whether the <code>next-gen-ecomm</code> feature is enabled. If
  393. it is then the emailer adds some extra order cancellation content to the
  394. email.</p>
  395. <p>While this looks like a reasonable approach, it's very brittle. The decision on
  396. whether to include order cancellation functionality in our invoice emails is wired
  397. directly to that rather broad <code>next-gen-ecomm</code> feature - using a magic string, no less. Why should
  398. the invoice emailling code need to know that the order cancellation content is
  399. part of the next-gen feature set? What happens if we'd like to turn on some parts
  400. of the next-gen functionality without exposing order cancellation? Or vice versa?
  401. What if we decide we'd like to only roll out order cancellation to certain users?
  402. It is quite common for these sort of "toggle scope" changes to occur as features
  403. are developed. Also bear in mind that these toggle points tend to proliferate
  404. throughout a codebase. With our current approach since the toggling decision logic
  405. is part of the toggle point any change to that decision logic will require
  406. trawling through all those toggle points which have spread through the
  407. codebase.</p>
  408. <p>Happily, <a href="https://en.wikipedia.org/wiki/Fundamental_theorem_of_software_engineering">any
  409. problem in software can be solved by adding a layer of indirection</a>. We can
  410. decouple a toggling decision point from the logic behind that decision like
  411. so:</p>
  412. <p class="code-label">featureDecisions.js
  413. </p>
  414. <pre class="code"> function createFeatureDecisions(features){
  415. return {
  416. includeOrderCancellationInEmail(){
  417. return features.isEnabled("next-gen-ecomm");
  418. }
  419. // ... additional decision functions also live here ...
  420. };
  421. }
  422. </pre>
  423. <p class="code-label">invoiceEmailer.js
  424. </p>
  425. <pre class="code"> const features = fetchFeatureTogglesFromSomewhere();
  426. const featureDecisions = createFeatureDecisions(features);
  427. function generateInvoiceEmail(){
  428. const baseEmail = buildEmailForInvoice(this.invoice);
  429. if( featureDecisions.includeOrderCancellationInEmail() ){
  430. return addOrderCancellationContentToEmail(baseEmail);
  431. }else{
  432. return baseEmail;
  433. }
  434. }
  435. </pre>
  436. <p>We've introduced a <code>FeatureDecisions</code> object, which acts as a collection point
  437. for any feature toggle decision logic. We create a decision method on this object
  438. for each specific toggling decision in our code - in this case "should we include
  439. order cancellation functionality in our invoice email" is represented by the
  440. <code>includeOrderCancellationInEmail</code> decision method. Right now the decision "logic"
  441. is a trivial pass-through to check the state of the <code>next-gen-ecomm</code> feature, but
  442. now as that logic evolves we have a singular place to manage it. Whenever we want
  443. to modify the logic of that specific toggling decision we have a single place to
  444. go. We might want to modify the scope of the decision - for example which specific
  445. feature flag controls the decision. Alternatively we might need to modify the
  446. reason for the decision - from being driven by a static toggle configuration to being
  447. driven by an A/B experiment, or by an operational concern such as an outage in
  448. some of our order cancellation infrastructure. In all cases our invoice emailer
  449. can remain blissfully unaware of how or why that toggling decision is being
  450. made.</p>
  451. </div>
  452. <div id="InversionOfDecision">
  453. <h3>Inversion of Decision</h3>
  454. <p>In the previous example our invoice emailer was responsible for asking the
  455. feature flagging infrastructure how it should perform. This means our invoice emailer has one
  456. extra concept it needs to be aware of - feature flagging - and an extra module it
  457. is coupled to. This makes the invoice emailer harder to work with and think about
  458. in isolation, including making it harder to test. As feature flagging has a
  459. tendency to become more and more prevalent in a system over time we will see more
  460. and more modules becoming coupled to the feature flagging system as a global
  461. dependency. Not the ideal scenario.</p>
  462. <p>In software design we can often solve these coupling issues by applying
  463. Inversion of Control. This is true in this case. Here's how we might decouple our
  464. invoice emailer from our feature flagging infrastructure:</p>
  465. <p class="code-label">invoiceEmailer.js
  466. </p>
  467. <pre class="code"> function createInvoiceEmailler(config){
  468. return {
  469. generateInvoiceEmail(){
  470. const baseEmail = buildEmailForInvoice(this.invoice);
  471. if( config.includeOrderCancellationInEmail ){
  472. return addOrderCancellationContentToEmail(email);
  473. }else{
  474. return baseEmail;
  475. }
  476. },
  477. // ... other invoice emailer methods ...
  478. };
  479. }</pre>
  480. <p class="code-label">featureAwareFactory.js
  481. </p>
  482. <pre class="code"> function createFeatureAwareFactoryBasedOn(featureDecisions){
  483. return {
  484. invoiceEmailler(){
  485. return createInvoiceEmailler({
  486. includeOrderCancellationInEmail: featureDecisions.includeOrderCancellationInEmail()
  487. });
  488. },
  489. // ... other factory methods ...
  490. };
  491. }</pre>
  492. <p>Now, rather than our <code>InvoiceEmailler</code> reaching out to <code>FeatureDecisions</code> it
  493. has those decisions injected into it at construction time via a <code>config</code> object.
  494. <code>InvoiceEmailler</code> now has no knowledge whatsoever about feature flagging. It just
  495. knows that some aspects of its behavior can be configured at runtime. This also
  496. makes testing <code>InvoiceEmailler</code>'s behavior easier - we can test the way that it
  497. generates emails both with and without order cancellation content just by passing
  498. a different configuration option during test:</p>
  499. <pre class="code">describe( 'invoice emailling', function(){
  500. it( 'includes order cancellation content when configured to do so', function(){
  501. // Given
  502. const emailler = createInvoiceEmailler({includeOrderCancellationInEmail:true});
  503. // When
  504. const email = emailler.generateInvoiceEmail();
  505. // Then
  506. verifyEmailContainsOrderCancellationContent(email);
  507. };
  508. it( 'does not includes order cancellation content when configured to not do so', function(){
  509. // Given
  510. const emailler = createInvoiceEmailler({includeOrderCancellationInEmail:false});
  511. // When
  512. const email = emailler.generateInvoiceEmail();
  513. // Then
  514. verifyEmailDoesNotContainOrderCancellationContent(email);
  515. };
  516. });
  517. </pre>
  518. <p>We also introduced a <code>FeatureAwareFactory</code> to centralize the creation of these
  519. decision-injected objects. This is an application of the general Dependency
  520. Injection pattern. If a DI system were in play in our codebase then we'd probably
  521. use that system to implement this approach.</p>
  522. </div>
  523. <div id="AvoidingConditionals">
  524. <h3>Avoiding conditionals</h3>
  525. <p>In our examples so far our Toggle Point has been implemented using an if
  526. statement. This might make sense for a simple, short-lived toggle. However point
  527. conditionals are not advised anywhere where a feature will require several Toggle Points, or
  528. where you expect the Toggle Point to be long-lived. A more maintainable
  529. alternative is to implement alternative codepaths using some sort of Strategy
  530. pattern:</p>
  531. <p class="code-label">invoiceEmailler.js
  532. </p>
  533. <pre class="code"> function createInvoiceEmailler(additionalContentEnhancer){
  534. return {
  535. generateInvoiceEmail(){
  536. const baseEmail = buildEmailForInvoice(this.invoice);
  537. return additionalContentEnhancer(baseEmail);
  538. },
  539. // ... other invoice emailer methods ...
  540. };
  541. }</pre>
  542. <p class="code-label">featureAwareFactory.js
  543. </p>
  544. <pre class="code"> function identityFn(x){ return x; }
  545. function createFeatureAwareFactoryBasedOn(featureDecisions){
  546. return {
  547. invoiceEmailler(){
  548. if( featureDecisions.includeOrderCancellationInEmail() ){
  549. return createInvoiceEmailler(addOrderCancellationContentToEmail);
  550. }else{
  551. return createInvoiceEmailler(identityFn);
  552. }
  553. },
  554. // ... other factory methods ...
  555. };
  556. }</pre>
  557. <p>Here we're applying a Strategy pattern by allowing our invoice emailer to be
  558. configured with a content enhancement function. <code>FeatureAwareFactory</code> selects a
  559. strategy when creating the invoice emailer, guided by its <code>FeatureDecision</code>. If
  560. order cancellation should be in the email it passes in an enhancer function which
  561. adds that content to the email. Otherwise it passes in an <code>identityFn</code> enhancer -
  562. one which has no effect and simply passes the email back without
  563. modifications.</p>
  564. </div>
  565. </div>
  566. <div id="ToggleConfiguration"><hr class="topSection"/>
  567. <h2>Toggle Configuration</h2>
  568. <div id="DynamicRoutingVsDynamicConfiguration">
  569. <h3>Dynamic routing vs dynamic configuration</h3>
  570. <p>Earlier we divided feature flags into those whose toggle routing decisions are
  571. essentially static for a given code deployment vs those whose decisions vary
  572. dynamically at runtime. It's important to note that there are two ways in which a
  573. flag's decisions might change at runtime. Firstly, something like a
  574. Ops Toggle might be dynamically <i>re-configured</i> from On to Off in response to a
  575. system outage. Secondly, some categories of toggles such as Permissioning Toggles
  576. and Experiment Toggles make a dynamic routing decision for each request based on
  577. some request context such as which user is making the request. The former is
  578. dynamic via re-configuration, while the later is inherently dynamic. These
  579. inherently dynamic toggles may make highly dynamic <b>decisions</b> but still have a
  580. <b>configuration</b> which is quite static, perhaps only changeable via
  581. re-deployment. Experiment Toggles are an example of this type of feature flag - we
  582. don't really need to be able to modify the parameters of an experiment at runtime.
  583. In fact doing so would likely make the experiment statistically invalid.</p>
  584. </div>
  585. <div id="PreferStaticConfiguration">
  586. <h3>Prefer static configuration</h3>
  587. <p>Managing toggle configuration via source control and re-deployments is
  588. preferable, if the nature of the feature flag allows it. Managing toggle configuration
  589. via source control gives us the same benefits that we get by using source control
  590. for things like infrastructure as code. It can allows toggle configuration
  591. to live alongside the codebase being toggled, which provides a really big win:
  592. toggle configuration will move through your Continuous Delivery pipeline in the
  593. exact same way as a code change or an infrastructure change would. This enables
  594. the full the benefits of CD - repeatable builds which are verified in a consistent
  595. way across environments. It also greatly reduces the testing burden of feature flags.
  596. There is less need to verify how the release will perform with both a toggle Off
  597. and On, since that state is baked into the release and won't be changed (for less
  598. dynamic flags at least). Another benefit of toggle configuration living
  599. side-by-side in source control is that we can easily see the state of the toggle
  600. in previous releases, and easily recreate previous releases if needed.</p>
  601. </div>
  602. <div id="ApproachesForManagingToggleConfiguration">
  603. <h3>Approaches for managing toggle configuration</h3>
  604. <p>While static configuration is preferable there are cases such as Ops Toggles where a more dynamic approach is required. Let's look at some options for managing toggle configuration, ranging from approaches which are simple but less dynamic
  605. through to some approaches which are highly sophisticated but come with a lot of
  606. additional complexity.</p>
  607. </div>
  608. <div id="HardcodedToggleConfiguration">
  609. <h3>Hardcoded Toggle Configuration</h3>
  610. <p>The most basic technique - perhaps so basic as to not be considered a Feature
  611. Flag - is to simply comment or uncomment blocks of code. For example:</p>
  612. <pre class="code">function reticulateSplines(){
  613. //return oldFashionedSplineReticulation();
  614. return enhancedSplineReticulation();
  615. }
  616. </pre>
  617. <p>Slightly more sophisticated than the commenting approach is the use of a
  618. preprocessor's <code>#ifdef</code> feature, where available.</p>
  619. <p>Because this type of hardcoding doesn't allow dynamic re-configuration of a
  620. toggle it is only suitable for feature flags where we're willing to follow a pattern of
  621. deploying code in order to re-configure the flag.</p>
  622. </div>
  623. <div id="ParameterizedToggleConfiguration">
  624. <h3>Parameterized Toggle Configuration</h3>
  625. <p>The build-time configuration provided by hardcoded configuration isn't flexible
  626. enough for many use cases, including a lot of testing scenarios. A simple approach which at least allows
  627. feature flags to be re-configured without re-building an app or service is to specify
  628. Toggle Configuration via command-line arguments or environment variables. This is
  629. a simple and time-honored approach to toggling which has been around since well
  630. before anyone referred to the technique as Feature Toggling or Feature Flagging. However it comes with
  631. limitations. It can become unwieldy to coordinate configuration across a large
  632. number of processes, and changes to a toggle's configuration require either a re-deploy or at the
  633. very least a process restart (and probably privileged access to servers by the
  634. person re-configuring the toggle too).</p>
  635. </div>
  636. <div id="ToggleConfigurationFile">
  637. <h3>Toggle Configuration File</h3>
  638. <p>Another option is to read Toggle Configuration from some sort of structured
  639. file. It's quite common for this approach to Toggle Configuration to begin life as
  640. one part of a more general application configuration file.</p>
  641. <p>With a Toggle Configuration file you can now re-configure a feature flag by simply
  642. changing that file rather than re-building application code itself. However,
  643. although you don't need to re-build your app to toggle a feature in most cases
  644. you'll probably still need to perform a re-deploy in order to re-configure a
  645. flag.</p>
  646. </div>
  647. <div id="ToggleConfigurationInAppDb">
  648. <h3>Toggle Configuration in App DB</h3>
  649. <p>Using static files to manage toggle configuration can become cumbersome once
  650. you reach a certain scale. Modifying configuration via files is relatively fiddly.
  651. Ensuring consistency across a fleet of servers becomes a challenge, making changes
  652. consistently even more so. In response to this many organizations move Toggle
  653. Configuration into some type of centralized store, often an existing application
  654. DB. This is usually accompanied by the build-out of some form of admin UI which
  655. allows system operators, testers and product managers to view and modify Features
  656. Flags and their configuration. </p>
  657. </div>
  658. <div id="DistributedToggleConfiguration">
  659. <h3>Distributed Toggle Configuration</h3>
  660. <p>Using a general purpose DB which is already part of the system architecture to
  661. store toggle configuration is very common; it's an obvious place to go once
  662. Feature Flags are introduced and start to gain traction. However nowadays there
  663. are a breed of special-purpose hierarchical key-value stores which are a better
  664. fit for managing application configuration - services like Zookeeper, etcd, or
  665. Consul. These services form a distributed cluster which provides a shared source
  666. of environmental configuration for all nodes attached to the cluster.
  667. Configuration can be modified dynamically whenever required, and all nodes in the
  668. cluster are automatically informed of the change - a very handy bonus feature.
  669. Managing Toggle Configuration using these systems means we can have Toggle Routers
  670. on each and every node in a fleet making decisions based on Toggle Configuration
  671. which is coordinated across the entire fleet. </p>
  672. <p>Some of these systems (such as Consul) come with an admin UI which provides a
  673. basic way to manage Toggle Configuration. However at some point a small custom app
  674. for administering toggle config is usually created.</p>
  675. </div>
  676. <div id="OverridingConfiguration">
  677. <h3>Overriding configuration</h3>
  678. <p>So far our discussion has assumed that all configuration is provided by a
  679. singular mechanism. The reality for many systems is more sophisticated, with
  680. overriding layers of configuration coming from various sources. With Toggle
  681. Configuration it's quite common to have a default configuration along with
  682. environment-specific overrides. Those overrides may come from something as simple
  683. as an additional configuration file or something sophisticated like a Zookeeper
  684. cluster. Be aware that any environment-specific overriding runs counter to the
  685. Continuous Delivery ideal of having the exact same bits and configuration flow all
  686. the way through your delivery pipeline. Often pragmatism dictates that some
  687. environment-specific overrides are used, but striving to keep both your deployable
  688. units and your configuration as environment-agnostic as possible will lead to a
  689. simpler, safer pipeline. We'll re-visit this topic shortly when we talk about
  690. testing a feature toggled system.</p>
  691. <div id="Per-requestOverrides">
  692. <h4>Per-request overrides</h4>
  693. <p>An alternative approach to a environment-specific configuration overrides is
  694. to allow a toggle's On/Off state to be overridden on a per-request basis by way
  695. of a special cookie, query parameter, or HTTP header. This has a few advantages
  696. over a full configuration override. If a service is load-balanced you can still
  697. be confident that the override will be applied no matter which service instance
  698. you are hitting. You can also override feature flags in a production environment
  699. without affecting other users, and you're less likely to accidentally leave an
  700. override in place. If the per-request override mechanism uses persistent cookies
  701. then someone testing your system can configure their own custom set of toggle
  702. overrides which will remain consistently applied in their browser. </p>
  703. <p>The downside of this per-request approach is that it introduces a risk that
  704. curious or malicious end-users may modify feature toggle state themselves. Some
  705. organizations may be uncomfortable with the idea that some unreleased features
  706. may be publicly accessible to a sufficiently determined party.
  707. Cryptographically signing your override configuration is one option to alleviate
  708. this concern, but regardless this approach will increase the complexity - and
  709. attack surface - of your feature toggling system.</p>
  710. <p>I elaborate on this technique for cookie-based overrides in <a href="http://blog.thepete.net/blog/2012/11/06/cookie-based-feature-flag-overrides/">this
  711. post</a> and have also <a href="http://blog.thepete.net/blog/2013/08/24/introducing-rack-flags/">described a
  712. ruby implementation</a> open-sourced by myself and a ThoughtWorks colleague.</p>
  713. </div>
  714. </div>
  715. </div>
  716. <div id="WorkingWithFeature-flaggedSystems"><hr class="topSection"/>
  717. <h2>Working with feature-flagged systems </h2>
  718. <p>While feature toggling is absolutely a helpful technique it does also bring
  719. additional complexity. There are a few techniques which can help make life easier
  720. when working with a feature-flagged system.</p>
  721. <div id="ExposeCurrentFeatureToggleConfiguration">
  722. <h3>Expose current feature toggle configuration</h3>
  723. <p>It's always been a helpful practice to embed build/version numbers into a
  724. deployed artifact and expose that metadata somewhere so that a dev, tester or operator can
  725. find out what specific code is running in a given environment. The same idea
  726. should be applied with feature flags. Any system using feature flags should
  727. expose some way for an operator to discover the current state of the toggle
  728. configuration. In an HTTP-oriented SOA system this is often accomplished via
  729. some sort of metadata API endpoint or endpoints. See for example Spring Boot's
  730. <a href="http://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html">Actuator
  731. endpoints</a>.</p>
  732. </div>
  733. <div id="TakeAdvantageOfStructuredToggleConfigurationFiles">
  734. <h3>Take advantage of structured Toggle Configuration files</h3>
  735. <p>It's typical to store base Toggle Configuration in some sort of structured,
  736. human-readable file (often in YAML format) managed via source-control. There are
  737. some additional benefits we can derive from this file. Including a
  738. human-readable description for each toggle is surprisingly useful, particularly
  739. for toggles managed by folks other than the core delivery team. What would you
  740. prefer to see when trying to decide whether to enable an Ops toggle
  741. during a production outage event: <b>basic-rec-algo</b> or <b>"Use a simplistic
  742. recommendation algorithm. This is fast and produces less load on backend
  743. systems, but is way less accurate than our standard algorithm."</b>? Some teams also
  744. opt to include additional metadata in their toggle configuration files such as a
  745. creation date, a primary developer contact, or even an expiration date for
  746. toggles which are intended to be short lived.</p>
  747. </div>
  748. <div id="ManageDifferentTogglesDifferently">
  749. <h3>Manage different toggles differently</h3>
  750. <p>As discussed earlier, there are various categories of Feature Toggles with
  751. different characteristics. These differences should be embraced, and different
  752. toggles managed in different ways, even if all the various toggles might
  753. be controlled using the same technical machinery. </p>
  754. <p>Let's revisit our previous example of an ecommerce site which has a
  755. Recommended Products section on the homepage. Initially we might have placed
  756. that section behind a Release Toggle while it was under development. We might
  757. then have moved it to being behind an Experiment Toggle to validate that it was
  758. helping drive revenue. Finally we might move it behind an Ops Toggle so that we
  759. can turn it off when we're under extreme load. If we've followed the earlier
  760. advice around de-coupling decision logic from Toggle Points then these
  761. differences in toggle category should have had no impact on the Toggle Point
  762. code at all. </p>
  763. <p>However from a feature flag management perspective these transitions
  764. absolutely should have an impact. As part of transitioning from Release Toggle
  765. to an Experiment Toggle the way the toggle is configured will change, and likely
  766. move to a different area - perhaps into an Admin UI rather than a yaml file in
  767. source control. Product folks will likely now manage the configuration rather
  768. than developers. Likewise, the transition from Experiment Toggle to Ops Toggle
  769. will mean another change in how the toggle is configured, where that
  770. configuration lives, and who manages the configuration.</p>
  771. </div>
  772. <div id="FeatureTogglesIntroduceValidationComplexity">
  773. <h3>Feature Toggles introduce validation complexity</h3>
  774. <p>With feature-flagged systems our Continuous Delivery process becomes more
  775. complex, particularly in regard to testing. We'll often need to test
  776. multiple codepaths for the same artifact as it moves through a CD pipeline. To
  777. illustrate why, imagine we are shipping a system which can either use a new
  778. optimized tax calculation algorithm if a toggle is on, or otherwise continue to
  779. use our existing algorithm. At the time that a given deployable artifact is
  780. moving through our CD pipeline we can't know whether the toggle will at some
  781. point be turned On or Off in production - that's the whole point of feature
  782. flags after all. Therefore in order to validate all codepaths which may end up
  783. live in production we must perform test our artifact in <b>both</b> states: with
  784. the toggle flipped On and flipped Off. </p>
  785. <div class="figure " id="feature-toggles-testing.png"><img src="feature-toggles/feature-toggles-testing.png"/>
  786. <p class="photoCaption"/>
  787. </div>
  788. <p class="clear"/>
  789. <p>We can see that with a single toggle in play this introduces a requirement to
  790. double up on at least some of our testing. With multiple toggles in play we have
  791. a combinatoric explosion of possible toggle states. Validating behavior for each
  792. of these states would be a monumental task. This can lead to some healthy
  793. skepticism towards Feature Flags from folks with a testing focus. </p>
  794. <p>Happily, the situation isn't as bad as some testers might initially imagine.
  795. While a feature-flagged release candidate does need testing with a few toggle
  796. configurations, it is not necessary to test *every* possible combination. Most
  797. feature flags will not interact with each other, and most releases will not
  798. involve a change to the configuration of more than one feature flag. </p>
  799. <div class="soundbite">
  800. <p>a good convention is to enable existing or legacy behavior when a Feature Flag is Off and new or future behavior when it's On.</p>
  801. </div>
  802. <p>So, which feature toggle configurations should a team test? It's most
  803. important to test the toggle configuration which you expect to become live in
  804. production, which means the current production toggle configuration plus any
  805. toggles which you intend to release flipped On. It's also wise to test the
  806. fall-back configuration where those toggles you intend to release are also
  807. flipped Off. To avoid any surprise regressions in a future release many teams
  808. also perform some tests with all toggles flipped On. Note that this advice only
  809. makes sense if you stick to a convention of toggle semantics where existing or
  810. legacy behavior is enabled when a feature is Off and new or future behavior is
  811. enabled when a feature is On.</p>
  812. <p>If your feature flag system doesn't support runtime configuration then you
  813. may have to restart the process you're testing in order to flip a toggle, or
  814. worse re-deploy an artifact into a testing environment. This can have a very
  815. detrimental effect on the cycle time of your validation process, which in turn
  816. impacts the all important feedback loop that CI/CD provides. To avoid this issue
  817. consider exposing an endpoint which allows for dynamic in-memory
  818. re-configuration of a feature flag. These types of override becomes even more
  819. necessary when you are using things like Experiment Toggles where it's even more
  820. fiddly to exercise both paths of a toggle.</p>
  821. <p>This ability to dynamically re-configure specific service instances is a very
  822. sharp tool. If used inappropriately it can cause a lot of pain and confusion
  823. in a shared environment. This facility should only ever be used by automated
  824. tests, and possibly as part of manual exploratory testing and debugging. If
  825. there is a need for a more general-purpose toggle control mechanism for use in
  826. production environments it would be best built out using a real distributed
  827. configuration system as discussed in the Toggle Configuration section above.</p>
  828. </div>
  829. <div id="WhereToPlaceYourToggle">
  830. <h3>Where to place your toggle</h3>
  831. <div id="TogglesAtTheEdge">
  832. <h4>Toggles at the edge</h4>
  833. <p>For categories of toggle which need per-request context (Experiment
  834. Toggles, Permissioning Toggles) it makes sense to place Toggle Points in the
  835. edge services of your system - i.e. the publicly exposed web apps that present
  836. functionality to end users. This is where your user's individual requests
  837. first enter your domain and thus where your Toggle Router has the most context
  838. available to make toggling decisions based on the user and their request. A
  839. side-benefit of placing Toggle Points at the edge of your system is that it
  840. keeps fiddly conditional toggling logic out of the core of your system. In
  841. many cases you can place your Toggle Point right where you're rendering HTML,
  842. as in this Rails example:</p>
  843. <p class="code-label">someFile.erb
  844. </p>
  845. <pre class="code"> &lt;%= if featureDecisions.showRecommendationsSection? %&gt;
  846. &lt;%= render 'recommendations_section' %&gt;
  847. &lt;% end %&gt;</pre>
  848. <p>Placing Toggle Points at the edges also makes sense when you are controlling access
  849. to new user-facing features which aren't yet ready for launch. In this context you can
  850. again control access using a toggle which simply shows or hides UI elements. As an
  851. example, perhaps you are building the ability to <a href="https://developers.facebook.com/docs/facebook-login">log in to your application using
  852. Facebook</a> but aren't ready to roll it out to users just yet. The implementation of
  853. this feature may involve changes in various parts of your architecture, but you can
  854. control exposure of the feature with a simple feature toggle at the UI layer which
  855. hides the "Log in with Facebook" button.</p>
  856. <p>It's interesting to note that with some of
  857. these types of feature flag the bulk of the unreleased functionality itself might
  858. actually be publicly exposed, but sitting at a url which is not discoverable by
  859. users.</p>
  860. </div>
  861. <div id="TogglesInTheCore">
  862. <h4>Toggles in the core </h4>
  863. <p>There are other types of lower-level toggle which must be placed deeper
  864. within your architecture. These toggles are usually technical in nature, and
  865. control how some functionality is implemented internally. An example would be a
  866. Release Toggle which controls whether to use a new piece of caching
  867. infrastructure in front of a third-party API or just route requests directly to
  868. that API. Localizing these toggling decisions within the service whose
  869. functionality is being toggled is the only sensible option in these cases.</p>
  870. </div>
  871. </div>
  872. <div id="ManagingTheCarryingCostOfFeatureToggles">
  873. <h3>Managing the carrying cost of Feature Toggles</h3>
  874. <p>Feature Flags have a tendency to multiply rapidly, particularly when first
  875. introduced. They are useful and cheap to create and so often a lot are created.
  876. However toggles do come with a carrying cost. They require you to introduce new
  877. abstractions or conditional logic into your code. They also introduce a
  878. significant testing burden. Knight Capital Group's <a href="http://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/">$460 million dollar
  879. mistake</a>
  880. serves as a cautionary tale on what can go wrong when you don't manage your
  881. feature flags correctly (amongst other things).</p>
  882. <div class="soundbite">
  883. <p>Savvy teams view their Feature Toggles as inventory which comes with a carrying cost, and work to keep that inventory as low as possible.</p>
  884. </div>
  885. <p>Savvy teams view the Feature Toggles in their codebase as inventory which comes
  886. with a carrying cost and seek to keep that inventory as low as possible.
  887. In order to keep the number of feature flags manageable a team
  888. must be proactive in removing feature flags that are no longer needed. Some
  889. teams have a rule of always adding a toggle removal task onto the team's backlog
  890. whenever a Release Toggle is first introduced. Other teams put "expiration dates"
  891. on their toggles. Some go as far as creating "time bombs" which will fail a test
  892. (or even refuse to start an application!) if a feature flag is still around after its
  893. expiration date. We can also apply a Lean approach to reducing inventory, placing
  894. a limit on the number of feature flags a system is allowed to have at any one time. Once
  895. that limit is reached if someone wants to add a new toggle they will first need to
  896. do the work to remove an existing flag.</p>
  897. </div>
  898. </div>