davidbgk
/
larlet-fr-david-cache
Следить 1
В избранное 0
Форкнуть 0
Код Релизы 0 Активность
A place to cache linked articles (think custom and personal wayback machine)
Вы не можете выбрать более 25 тем Темы должны начинаться с буквы или цифры, могут содержать дефисы(-) и должны содержать не более 35 символов.
217 коммитов
1 ветка
Дерево: ca671a43be
Ветки Теги
${ item.name }
Создать ветку ${ searchTerm }
из 'ca671a43be'
${ noResults }
larlet-fr-david-cache/cache/e17eedd1ba5b82341c39695cd3c.../index.md

index.md 36KB
Исходник Normal View История

Add cache data
4 лет назад
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133 
  1. title: Experiments, growth engineering, and exposing company secrets through your API: Part 1

  2. url: https://blog.jonlu.ca/posts/experiments-and-growth-hacking

  3. hash_url: e17eedd1ba5b82341c39695cd3c84723

  4. 

  5. <p><i>This is the first part of a two part series on modern testing infrastructure and methodologies. Part 2 will focus on an analysis of these companies’ testing habits - how often they introduce new tests, the methodology and thought process behind their tests, and any other information we may be able to glean from their testing configurations.</i></p><p>Most medium to large companies now runs <a href="https://en.wikipedia.org/wiki/A/B_testing" target="_blank" rel="noopener noreferrer">A/B tests</a> and new feature experiments on segments of their user base. They are a great way to check whether a feature will have long time success, and get observable metrics on the repercussion of their changes.</p><p>I wanted to explore the methodology with which a few large companies implement their testing, as well as see if there was any potentially confidential information leakage. The actual implementation of these tests are often siloed - due to the recent growth of single page apps and separation of concerns, they rely on the <em>front end</em> or <em>client</em> to make the differentiation of tests, rather than making these decisions server side.</p><p>In practice this means that a lot of companies have private API routes that define their testing infrastructure. The client will request the current tests enabled for the account, and server will reply back, often with a huge list of <strong>all</strong> the tests the company is running. Below I explored how some of the most popular sites do it.</p><h3 id="lyft">Lyft</h3><p>When you sign in to Lyft on the web client, one of the network requests fired is to <code class="highlighter-rouge">https://www.lyft.com/api/experiments/configurations</code>. This replies back with a 180kb JSON object with every experiment they are running, as well as the user’s current status.</p><picture class="centered-image"><source srcset="/images/lyft-api-network.webp" type="image/webp"></source><source srcset="/images/lyft-api-network.png" type="image/png"></source><img alt="Lyft experiments route" class="centered-image" src="/images/lyft-api-network.png"></picture><p class="footnote">Lyft experiments route</p><p>You can see the full object <a href="https://pastebin.com/01w9yG4L" target="_blank" rel="noopener noreferrer">here</a>. There’s too much information here to unpack all of it, but we can go over some of the most interesting tidbits.</p><p>There are two main keys - <code class="highlighter-rouge">assignments</code> and <code class="highlighter-rouge">variables</code>. Assignments seem to be the ones with testing - each key has you in one of their predefined groups. These seem to be:</p><ul><li>Control</li><li>Static</li><li>Treatment</li><li>Variant_# (where # is what variant of the experiment you are receiving)</li></ul><p>Additionally, a significant number of experiments are prefixed with <code class="highlighter-rouge">Justin</code> or <code class="highlighter-rouge">Micha</code> - I’m not sure if these are the names of internal testing tools or employees that are prefixing their tests with their names.</p><p>While some of the experiment names are inscrutable (<code class="highlighter-rouge">exodus ETE test2: "control"</code>, <code class="highlighter-rouge">MichaTntTest_e89f281d-53bb-4b2b-b1b9-f28a840bb75b: "treatment"</code>, and <code class="highlighter-rouge">EliuTest</code> could mean anything), others offer a glimpse into the direction that Lyft is currently exploring. These are all guesses based on key names, but they seem to be specific enough that we can make some inference.</p><ul><li><code class="highlighter-rouge">PPTacticalPriceSFOV4</code> - Tactical pricing at SFO - they’re differentiating their pricing to SFO. Airports are a really common occurrence in their testing names.</li><li><code class="highlighter-rouge">PXCPaxSREarlyArrival5minWaitFromAcceptV2</code> with a value of <code class="highlighter-rouge">"300s_from_accept"</code> - How long a driver should wait after accepting?</li><li><code class="highlighter-rouge">PXCPaxCancelFrictionAcceptanceEmpathy</code> - When canceling a ride, it looks like they might be changing the wording to modify user behavior, and perhaps lower cancel rates</li></ul><p>The next key, <code class="highlighter-rouge">variables</code>, seem to be mostly boolean settings on your user profile. There are currently 61 (!!) categories of variables.</p><p>These show your status as both a rider and a driver, and provides a <em>lot</em> of insight into what Lyft thinks you are. There are keys for checking if you’re excluded from certain promotions, whether they think you are abusing their system, and booleans that designate you as a Lyft employee/admin. Some of the most interesting ones are:</p><ul><li><code class="highlighter-rouge">lastmile.enableIncentiveZones</code> - This is false for me. Could either be for drivers to go to populated areas, or as incentive zones for specific users?</li><li><code class="highlighter-rouge">lastmile.userLegacyMap</code> - Are they using different maps? Could they be transitioning off of Google Maps?</li><li><code class="highlighter-rouge">payments.allow_amex_on_fd</code> - Based on other keys, <code class="highlighter-rouge">fd</code> means first data. This is false for me - maybe they don’t show Amex’s first because of the higher interchange fees on their end, so they’d prefer riders/drivers to use Visa/Mastercard?</li><li><code class="highlighter-rouge">payment.intuitAffiliateCode</code> - For me, this value is <code class="highlighter-rouge">lyftplat18</code>. Perhaps this is a promo code from intuit? It’s tax season right now, so maybe they’re gearing up for a partnership on TurboTax, from Intuit?</li><li><code class="highlighter-rouge">pricing.tpa</code> - There’s a whole section on <code class="highlighter-rouge">TPA</code>, which seems to stand for “Tactical Price Adjustment”; They’re pretty clearly aware of people checking between Uber and Lyft for every ride, and they have different tiers and values, maybe to just barely beat their competitors</li><li><code class="highlighter-rouge">api.lnHasStrictEarlyCancelFee</code> - This is false for me, and if I had to guess this seems like a penalty for people who often cancel their rides after they’ve been matched with a driver.</li><li><code class="highlighter-rouge">cancels</code> - There’s a whole section on “prediction” with the key <code class="highlighter-rouge">predictorEnabled</code>. For instance, for my account <code class="highlighter-rouge">pxcPaxNoShowTa1kPredictor</code> is <code class="highlighter-rouge">0.52</code> and <code class="highlighter-rouge">pxcPaxNoShowTa1kPredictorBkn</code> is <code class="highlighter-rouge">0.66</code>. This might be a likelihood of someone not showing up? I’m normally very respectful of my drivers and almost always am where I need to be before they show up, though, so I’m not sure if <code class="highlighter-rouge">0.52</code> is 52% or measuring some other metric.</li><li><code class="highlighter-rouge">enterprise.businessRewardsMapBanner</code> - Uber just recently announced Uber Rewards. Maybe they’re gearing up for a business version of this, with rewards when enterprise customers use them for their ride shares?</li><li><code class="highlighter-rouge">publicapi.ridetypes.courier.features.supportsWalking</code> - Uber recently rolled out “Express Pool”, which has you walk a small distance in exchange for a cheaper price. Maybe Lyft is trying the same approach?</li><li><code class="highlighter-rouge">service.venues.snap_to_destination_vegas_enabled</code> - A lot of the locations mention specific places. One of the keys is even <code class="highlighter-rouge">MacArthurBARTEnabled</code> - one of the BART stops in East Bay!</li><li><code class="highlighter-rouge">ProjectX</code> - This was one of the most intriguing keys. I asked a friend at Lyft and apparently it was the code name for the recent redesign!</li></ul><p>There also seems to be very little cleaning or filtering on Lyft’s end - for instance, these are some of the entries in <code class="highlighter-rouge">variables</code>:</p><div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">

  6.     </span><span class="s2">"paultest"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">

  7.         </span><span class="s2">"litetestvar"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">

  8.         </span><span class="s2">"foobar"</span><span class="p">:</span><span class="w"> </span><span class="s2">"This is fine"</span><span class="p">,</span><span class="w">

  9.         </span><span class="s2">"something"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">

  10.             </span><span class="s2">"foobar"</span><span class="p">:</span><span class="w"> </span><span class="s2">"This is fine"</span><span class="p">,</span><span class="w">

  11.             </span><span class="s2">"deploytest"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">

  12.         </span><span class="p">},</span><span class="w">

  13.         </span><span class="s2">"deploytesting"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">

  14.             </span><span class="s2">"deploytestfoo"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">

  15.         </span><span class="p">}</span><span class="w">

  16.     </span><span class="p">}</span><span class="w">

  17. </span><span class="p">}</span><span class="w">

  18. </span></code></pre></div></div><p>and</p><div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">

  19.     </span><span class="s2">"whatever"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">

  20.         </span><span class="s2">"paultest33"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">

  21.         </span><span class="s2">"paultestagain"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">

  22.     </span><span class="p">}</span><span class="w">

  23. </span><span class="p">}</span><span class="w">

  24. </span></code></pre></div></div><p>Overall we were able to glean a lot about their current strategy, just by looking at their experiment names. An interesting test would be to set up something like Burp Suite and replace the variables from false to true and vice versa, and see if you can unlock any hidden panels in the app.</p><p>The above is less than 10% of their total experiments - I invite you to look through the full experiments <a href="https://pastebin.com/01w9yG4L" target="_blank" rel="noopener noreferrer">here</a>, and let me know what you find!</p><p>Next I decided to intercept the experiments from the mobile app. This includes quite a bit more information. You can see this <a href="https://pastebin.com/ReRwPv0y" target="_blank" rel="noopener noreferrer">here</a>.</p><p>One of the most interesting:</p><ul><li><code class="highlighter-rouge">vars.client.autonomous</code> - It looks like Lyft is experimenting with Autonomous rides. For instance, the <code class="highlighter-rouge">console_far_stops</code> has Lyft Level 5 HQ and the Caltrain AV Stop, which seems to be Caltrain Autonomous Vehicle stop.</li></ul><p>There are 1,449 keys <em>just in vars.client</em> - this doesn’t include the 162 experiments being run in <code class="highlighter-rouge">variants</code>, or the 44 in <code class="highlighter-rouge">holdout_variants</code>. Lyft is running a lot of experiments!</p><p>You can see all the information <a href="https://pastebin.com/ReRwPv0y" target="_blank" rel="noopener noreferrer">here</a> and run your own analysis.</p><h3 id="airbnb">Airbnb</h3><p>Airbnb’s iOS app makes a request to <code class="highlighter-rouge">https://api.airbnb.com/v2/mario_experiments?client=ios</code>.</p><p>The first thing you notice is that there’s a URL parameter for the client. Putting in <code class="highlighter-rouge">ios</code> returns a large JSON object that’s about 33kb large. If you switch it to <code class="highlighter-rouge">android</code>, it replies back with an object that’s 44kb - apparently they’re running a lot more experiments on Android!</p><p>The iOS response can be found <a href="https://pastebin.com/zX1YfqKz" target="_blank" rel="noopener noreferrer">here</a> and the Android response can be found <a href="https://pastebin.com/DaieuA6e" target="_blank" rel="noopener noreferrer">here</a>.</p><p>The testing framework Airbnb seems to use is called Mario. Each of their experiments follows a similar format:</p><div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">

  25.   </span><span class="s2">"assigned_treatment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"static"</span><span class="p">,</span><span class="w">

  26.   </span><span class="s2">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"same translation as the one in mario_android group"</span><span class="p">,</span><span class="w">

  27.   </span><span class="s2">"experiment_name"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">

  28.   </span><span class="s2">"group_name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"mario_ios"</span><span class="p">,</span><span class="w">

  29.   </span><span class="s2">"locale"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">

  30.   </span><span class="s2">"parameter_name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"registraiton_log_in_menu"</span><span class="p">,</span><span class="w">

  31.   </span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"multi_language"</span><span class="p">,</span><span class="w">

  32.   </span><span class="s2">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Log in"</span><span class="p">,</span><span class="w">

  33.   </span><span class="s2">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"v0"</span><span class="w">

  34. </span><span class="p">},</span><span class="w">

  35. </span></code></pre></div></div><p>The great part is that there’s no guesswork involved - the “description” key tells us exactly what each experiment is.</p><p>Most of their experiments are focused around referrals and support. There didn’t seem to be anything particularly indicative of future direction - most of it was different wordings for referrals, different button placement, and various changes to the chat support system.</p><p>Airbnb also makes a request to <code class="highlighter-rouge">https://api.airbnb.com/v2/client_configs</code> - although this seems to be less experiments and more just general app config, it includes some interesting tidbits as well.</p><p>Some client configs are:</p><div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">

  36.   </span><span class="s2">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ios.block_dates_on_preapproval"</span><span class="p">,</span><span class="w">

  37.   </span><span class="s2">"launch"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">

  38. </span><span class="p">},</span><span class="w">

  39. </span><span class="p">{</span><span class="w">

  40.   </span><span class="s2">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"china_fee_launched"</span><span class="p">,</span><span class="w">

  41.   </span><span class="s2">"launch"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">

  42. </span><span class="p">},</span><span class="w">

  43. </span></code></pre></div></div><p>There’s a <em>lot</em> of references to China - it seems like a major focus of Airbnb right now is expansion in the Asian market.</p><p>The full client config can be found <a href="https://pastebin.com/5uaniYnn" target="_blank" rel="noopener noreferrer">here</a>. There’s a ton of experiments in there - 923 in the <code class="highlighter-rouge">experiments_assignments</code> category.</p><p>There seems to be a lot of new versions of existing parts of the app, such as “new_quick_pay”, “new_guest_inbox”, and “new_verifications_profile_completion”, a lot of which are not yet in the “launched” state.</p><p>Also as an aside, my work section for Airbnb used to say “Software Engineer at Google!” - but now it says “Software Engineer at (Hidden by Airbnb)”, and if I try to make it say “Google” again, the HTTP <code class="highlighter-rouge">PATCH</code> request replies with:</p><div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">

  44.   </span><span class="s2">"error_code"</span><span class="p">:</span><span class="w"> </span><span class="mi">400</span><span class="p">,</span><span class="w">

  45.   </span><span class="s2">"error_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"validation"</span><span class="p">,</span><span class="w">

  46.   </span><span class="s2">"error_message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"There was an error processing the request."</span><span class="p">,</span><span class="w">

  47.   </span><span class="s2">"error_details"</span><span class="p">:</span><span class="w"> </span><span class="s2">"the contents of the work field are disallowed"</span><span class="p">,</span><span class="w">

  48.   </span><span class="s2">"error_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"&lt;redacted&gt;"</span><span class="w">

  49. </span><span class="p">}</span><span class="w">

  50. </span></code></pre></div></div><p>They keep replacing “Google” with “(Hidden by Airbnb)” - I wonder if this is them trying to discourage users from saying “Google my name to get in contact with me directly to circumvent paying Airbnb’s fees”, or if they just have some feud with Google and want to limit their employees from using the app.</p><h3 id="pinterest">Pinterest</h3><p>Pinterest is currently running one of the most comprehensive experiment campaigns I’ve ever seen. Upon initial load of the app it makes a request to <code class="highlighter-rouge">https://api.pinterest.com/v3/users/me/</code>. This replies back with the full user object, as well as the <code class="highlighter-rouge">gatekeeper_experiments</code>.</p><p>The full experiment list is <a href="https://pastebin.com/3jZPazXs" target="_blank" rel="noopener noreferrer">here</a>. There are currently 823 active experiments. Some of the most interesting are as follows:</p><ul><li><code class="highlighter-rouge">ios_skin_tone_filter</code> - You can now filter for make up products by your skin tone!</li><li><code class="highlighter-rouge">ios_use_cases_board_ideas_quiz</code> - Quiz on different boards?</li><li><code class="highlighter-rouge">activation_male_topic_image_replacement</code> - Gendered content differentiation?</li></ul><picture class="centered-image"><source srcset="/images/burp-pt.webp" type="image/webp"></source><source srcset="/images/burp-pt.png" type="image/png"></source><img alt="Pinterest experiments route" class="centered-image" src="/images/burp-pt.png"></picture><p class="footnote">Burp Suite intercepting Pinterest API calls</p><h3 id="reddit">Reddit</h3><p>Reddit has been making huge changes over the past year - the reddit redesign was a big change for the company, and their leadership really seems to be trying to turn reddit into a social media app. On load of the app, it requests <code class="highlighter-rouge">https://gateway.reddit.com/redditmobile/1/ios/config</code>. This config has all the experiments currently enabled. The full reply can be found <a href="https://pastebin.com/TqhVdBkp" target="_blank" rel="noopener noreferrer">here</a>.</p><p>They currently have 19 “buckets”, which have corresponding “experiments”.</p><p>A lot have to do with content discovery and reddit recommendations, such as <code class="highlighter-rouge">best_of_community_carousel</code> and <code class="highlighter-rouge">trending_users_carousel</code>.</p><p>This lends credence to them pushing for user and company based followings, rather than specific subreddits.</p><h3 id="slack">Slack</h3><p>Slack has a <em>ton</em> of experiments going on.</p><p>Upon load of the app, it requests 3 experiments routes.</p><ul><li><code class="highlighter-rouge">https://slack.com/api/experiments.getEZFeatures</code></li><li><code class="highlighter-rouge">https://slack.com/api/experiments.getByUser</code></li><li><code class="highlighter-rouge">https://slack.com/api/experiments.getByVisitor</code></li></ul><p>They all follow a similar format, which is as follows:</p><div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">

  51.   </span><span class="s2">"experiment_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"482395575254"</span><span class="p">,</span><span class="w">

  52.   </span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"team"</span><span class="p">,</span><span class="w">

  53.   </span><span class="s2">"group"</span><span class="p">:</span><span class="w"> </span><span class="s2">"treatment"</span><span class="p">,</span><span class="w">

  54.   </span><span class="s2">"trigger"</span><span class="p">:</span><span class="w"> </span><span class="s2">"finished"</span><span class="p">,</span><span class="w">

  55.   </span><span class="s2">"log_exposures"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">

  56.   </span><span class="s2">"exposure_id"</span><span class="p">:</span><span class="w"> </span><span class="mi">54886002322</span><span class="w">

  57. </span><span class="p">}</span><span class="w">

  58. </span></code></pre></div></div><p>A lot of these have <code class="highlighter-rouge">"trigger": "finished"</code>, which might indicate that the experiment is over.</p><p>Some of the interesting ones are:</p><ul><li><code class="highlighter-rouge">stripe_l3_data</code></li><li><code class="highlighter-rouge">unified_autocomplete</code></li><li><code class="highlighter-rouge">ios_poseidon</code></li><li><code class="highlighter-rouge">monetization_no_flannel_login</code><h3 id="amazon">Amazon</h3></li></ul><p>Upon launch of the iOS Amazon app, it makes a request to <code class="highlighter-rouge">https://msh.amazon.com/mwl/assignments/v2</code> - this replies back with a large object containing each of the “Assignments”. The full reply can be found <a href="https://pastebin.com/bUkZ61un" target="_blank" rel="noopener noreferrer">here</a>.</p><p>Some of the most interesting ones are as these.</p><div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s2">"A9VS_AUGMENTED_REALITY_VIEW_DEEPLINK_129451"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">

  59.   </span><span class="s2">"AllocationVersion"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">

  60.   </span><span class="s2">"MayTrigger"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">

  61.   </span><span class="s2">"Treatment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"T1"</span><span class="w">

  62. </span><span class="p">},</span><span class="w">

  63. </span></code></pre></div></div><p>Amazon seems to be investing in AR, possibly to view items in augmented reality before you purchase them.</p><div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s2">"AMAZON_PRINTS_LAUNCH_PHOTOBOOK_191331"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">

  64.   </span><span class="s2">"AllocationVersion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ATVPDKIKX0DER--C:1"</span><span class="p">,</span><span class="w">

  65.   </span><span class="s2">"MayTrigger"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">

  66.   </span><span class="s2">"Treatment"</span><span class="p">:</span><span class="w"> </span><span class="s2">"C"</span><span class="w">

  67. </span><span class="p">},</span><span class="w">

  68. </span></code></pre></div></div><p>Amazon photobooks creates photobooks from your pictures?</p><p>There’s a lot of different experiments in there to unpack, although most of the others seemed fairly uninteresting.</p><h3 id="tinder">Tinder</h3><p>Dating apps as a whole are a really interesting place for AB testing due to the way users attitude towards them shifts. Experiments in dating apps could have a much more profound effect than most other sites.</p><p>Unfortunately I didn’t see anything that resembled experiments behind the scenes at Tinder. There were plenty of feature flags, like whether Tinder Groups were enabled, whether Tinder Places were enabled, or whether you were part of the “Tinder Elite”, the Tinder program for famous people.</p><p>While I wasn’t able to find the Tinder Experiments API route, I did want to include a mention here anyways. A few months back Tinder removed the ability to view a persons instagram - you could still see the photos but there was no link to the account or their username.</p><p>It turns out they didn’t update their API though - their API still replies back with your matches Instagram handle, so you can find them that way. It would probably be fairly trivial to override <code class="highlighter-rouge">fetch</code> on the web client to intercept all matches and display their Instagram with a Chrome Extension.</p><p>A sample profile is below.</p><div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">

  69.   </span><span class="s2">"_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"&lt;id&gt;"</span><span class="p">,</span><span class="w">

  70.   </span><span class="s2">"age_filter_max"</span><span class="p">:</span><span class="w"> </span><span class="mi">31</span><span class="p">,</span><span class="w">

  71.   </span><span class="s2">"age_filter_min"</span><span class="p">:</span><span class="w"> </span><span class="mi">18</span><span class="p">,</span><span class="w">

  72.   </span><span class="s2">"bio"</span><span class="p">:</span><span class="w"> </span><span class="s2">"&lt;bio&gt;"</span><span class="p">,</span><span class="w">

  73.   </span><span class="s2">"birth_date"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1996-02-26T23:47:24.858Z"</span><span class="p">,</span><span class="w">

  74.   </span><span class="s2">"create_date"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2018-04-12T07:07:23.169Z"</span><span class="p">,</span><span class="w">

  75.   </span><span class="s2">"crm_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"&lt;redacted&gt;"</span><span class="p">,</span><span class="w">

  76.   </span><span class="s2">"discoverable"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">

  77.   </span><span class="s2">"interests"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">

  78.     </span><span class="s2">"Facebook Likes, Redacted"</span><span class="w">

  79.   </span><span class="p">],</span><span class="w">

  80.   </span><span class="s2">"distance_filter"</span><span class="p">:</span><span class="w"> </span><span class="mi">100</span><span class="p">,</span><span class="w">

  81.   </span><span class="s2">"gender"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w">

  82.   </span><span class="s2">"gender_filter"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w">

  83.   </span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"JonLuca"</span><span class="p">,</span><span class="w">

  84.   </span><span class="s2">"photos"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">

  85.    </span><span class="s2">"Photos Array, Redacted"</span><span class="w">

  86.   </span><span class="p">],</span><span class="w">

  87.   </span><span class="s2">"photos_processing"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">

  88.   </span><span class="s2">"photo_optimizer_enabled"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">

  89.   </span><span class="s2">"photo_optimizer_has_result"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">

  90.   </span><span class="s2">"ping_time"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2019-02-23T23:47:24.301Z"</span><span class="p">,</span><span class="w">

  91.   </span><span class="s2">"jobs"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">

  92.     </span><span class="p">{</span><span class="w">

  93.       </span><span class="s2">"company"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">

  94.         </span><span class="s2">"displayed"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">

  95.         </span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Google"</span><span class="w">

  96.       </span><span class="p">},</span><span class="w">

  97.       </span><span class="s2">"title"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">

  98.         </span><span class="s2">"displayed"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">

  99.         </span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Engineer"</span><span class="w">

  100.       </span><span class="p">}</span><span class="w">

  101.     </span><span class="p">}</span><span class="w">

  102.   </span><span class="p">],</span><span class="w">

  103.   </span><span class="s2">"schools"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">

  104.     </span><span class="p">{</span><span class="w">

  105.       </span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"USC"</span><span class="p">,</span><span class="w">

  106.       </span><span class="s2">"displayed"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">

  107.     </span><span class="p">}</span><span class="w">

  108.   </span><span class="p">],</span><span class="w">

  109.   </span><span class="s2">"phone_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"&lt;redacted&gt;"</span><span class="p">,</span><span class="w">

  110.   </span><span class="s2">"interested_in"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">

  111.     </span><span class="mi">1</span><span class="w">

  112.   </span><span class="p">],</span><span class="w">

  113.   </span><span class="s2">"pos"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">

  114.     </span><span class="s2">"lat"</span><span class="p">:</span><span class="w"> </span><span class="mf">37.86871654555783</span><span class="p">,</span><span class="w">

  115.     </span><span class="s2">"lon"</span><span class="p">:</span><span class="w"> </span><span class="mf">-122.25116702766258</span><span class="w">

  116.   </span><span class="p">},</span><span class="w">

  117.   </span><span class="s2">"places_enabled"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">

  118.   </span><span class="s2">"autoplay_video"</span><span class="p">:</span><span class="w"> </span><span class="s2">"always"</span><span class="p">,</span><span class="w">

  119.   </span><span class="s2">"top_picks_discoverable"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">

  120.   </span><span class="s2">"photo_tagging_enabled"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">

  121. </span><span class="p">},</span><span class="w">

  122. </span><span class="p">{</span><span class="w">

  123.   </span><span class="s2">"username"</span><span class="p">:</span><span class="w"> </span><span class="s2">"&lt;my instagram username, redacted&gt;"</span><span class="p">,</span><span class="w">

  124.   </span><span class="s2">"profile_picture"</span><span class="p">:</span><span class="w"> </span><span class="s2">"&lt;url to profile pic&gt;"</span><span class="p">,</span><span class="w">

  125.   </span><span class="s2">"media_count"</span><span class="p">:</span><span class="w"> </span><span class="mi">33</span><span class="p">,</span><span class="w">

  126.   </span><span class="s2">"last_fetch_time"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2019-02-22T21:16:58.696Z"</span><span class="p">,</span><span class="w">

  127.   </span><span class="s2">"completed_initial_fetch"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">

  128.   </span><span class="s2">"photos"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">

  129.     </span><span class="s2">"&lt;photos array, redacted&gt;"</span><span class="w">

  130.   </span><span class="p">],</span><span class="w">

  131.   </span><span class="s2">"should_reauthenticate"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">

  132. </span><span class="p">}</span><span class="w">

  133. </span></code></pre></div></div><h3 id="facebook-and-instagram">Facebook and Instagram</h3><p>I wish I could’ve viewed Facebook and Instagram as well. Unfortunately they do SSL Stapling - any attempts at doing Burp Suite cert replacement fails, and no routes are shown. In the past, if you had a jailbroken phone, you could override the admin preferences and force show all experiments, with toggles and state.</p><p>Unfortunately I have an iPhone XS Max on 12.1.4, which doesn’t have a jailbreak. Once a jailbreak comes out I’ll be able to override TLS Stapling using a tool like <a href="https://github.com/iSECPartners/ios-ssl-kill-switch" target="_blank" rel="noopener noreferrer">iOS Kill Switch</a>, and we’ll be able to view the routes of all companies that do SSL Stapling (Snapchat, FB, Instagram, etc). This will hopefully be included in Part 2, which is set to come out within the next month or so.</p><h2 id="conclusion">Conclusion</h2><p>Most companies aren’t obfuscating or minimizing their experiment names, which leads to information leakage. This could prove dangerous in the future - if a company is slowly rolling out a new feature, it could give their competitors an advantage.</p><p>This is a common occurrence in the industry - nearly every company is siloing off their growth engineering department, which leads to siloed off experiment routes. This in turn makes it almost trivial to figure out what they’re working on, and make educated guesses at the 6 month roadmap of most tech services.</p><p>Some future companies I’d like to try and check out are Snapchat, Ebay, all the Google products and services, and LinkedIn.</p><p>There’s a lot more apps and services that this methodology works with. Feel free to reach out if you’re interested in finding any given companies experimentation campaigns.</p>