A place to cache linked articles (think custom and personal wayback machine)
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

2 年之前
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294
  1. title: Technical Solutions Poorly Solve Social Problems
  2. url: https://christine.website/blog/social-quandry-devops-2022-03-17
  3. hash_url: 3e8bb1b63246d6f97316864569492382
  4. <p class="conversation-chat">&lt;<b>Cadey</b>&gt; I just wanna lead this article out by saying that <em>I do not have all the
  5. answers here</em>. I really wish I did, but I also feel that I shouldn't have to
  6. have an answer in mind in order to raise a question. Please also keep in mind
  7. that this is coming from someone who has been working in devops for most of
  8. their career.</p>
  9. </div>
  10. <h2>Or: The Social Quandry of Devops</h2>
  11. <p>Technology is the cornerstone of our society. As a people we have seen the
  12. catalytic things that technology has enabled us to do. Through technology and
  13. new and innovative ways of applying it, we can help solve many problems. This
  14. leads some to envision technology as a panacea, a mythical cure-all that will
  15. make all our problems go away with the right use of it.</p>
  16. <p>This does not extend to social problems. Technical fixes for social problems are
  17. how we end up with an inadequate mess that can make the problem a lot worse than
  18. it was before. You've almost certainly been able to see this in action with
  19. social media (under the belief that allowing people to connect is so morally
  20. correct that it will bring in a new age of humanity that will be objectively
  21. good for everyone). The example I want to focus on today is the Devops
  22. philosophy. Devops is a technical solution (creating a new department) that
  23. helps work around social problems in workplaces (fundamental differences in
  24. priorities and end goals), and in the process it doesn't solve either very well.</p>
  25. <p>There are a lot of skillset paths that you can end up with in tech, but the two
  26. biggest ones are development (making the computer do new things) and systems
  27. administration (making computers keep doing those things). There are many other
  28. silos in the industry (technical writing, project/product management, etc.), but
  29. the two main ones are development and systems administration. These two groups
  30. have vastly different priorities, skillsets, needs and future goals, and as a
  31. result of this there is very little natural cross-pollenation between the two
  32. silos. I have seen this evolve into cultural resentment.</p>
  33. <div class="conversation">
  34. <p class="conversation-chat">&lt;<b>Cadey</b>&gt; Not to say that this phenomenon is exclusive to inter-department ties, I've
  35. also seen it happen intra-department over choice of programming language.</p>
  36. </div>
  37. <p>As far as the main differences go, development usually sees what could be. What
  38. new things could exist and what steps you need to take to get people there. This
  39. usually involves designing and implementing new software. The systems
  40. administration side of things is more likely to see it as a matter of
  41. integrating things into an existing whole, and then ensuring that whole is
  42. reliable and proven so they don't have to worry about it constantly. This causes
  43. a slower velocity forward and can result in extra process, slow momentum and
  44. stagnation. These two forces naturally come into conflict because they are
  45. vastly different things and have vastly different requirements and expectations.</p>
  46. <p>Development may want to use a new version of the compiler to support a language
  47. feature that will eliminate a lot of repetitive boilerplate. The sysadmins may
  48. not be able to ship that compiler in production build toolstack because of
  49. conflicting dependencies elsewhere, but they may also not want to ship that
  50. compiler because of fears over trusting unproven software in production.</p>
  51. <div class="conversation">
  52. <p class="conversation-chat">&lt;<b>Cadey</b>&gt; This fear sounds really odd at first glance, but this is a paraphrased version
  53. of a problem I actually encountered in the real world at one of my first big
  54. tech jobs. This place had some unique tech choices such as making their own fork
  55. of Ubuntu for "stability reasons", and the process to upgrade tools was a huge
  56. pain on the sysadmin side because it meant retesting and deploying a lot of
  57. internal tooling, which took a lot longer than the engineering team had patience
  58. for. This may not be the best example from a technical standpoint, but things
  59. don't have to make sense for them to exist.</p>
  60. </div>
  61. <p>This tension builds over a long period of time and can cause problems when the
  62. sysadmin team is chronically underfunded (due to the idea that they are
  63. successful when nothing goes wrong, also incurring the problem of success being
  64. a negative, which can make the sysadmin team look like a money pit when they are
  65. actually the very thing that is making the money generator generate money). This
  66. can also lead to avoidable burnout, unwarranted anxiety issues and unneeded
  67. suffering on both ends of the conflict.</p>
  68. <p>So given the unstoppable force of development and the immovable wall of
  69. sysadmin, an organizational compromise was made. This started out as many things
  70. with many names, but as the idea rippled throughout people's heads the name
  71. "devops" ended up sticking. Devops is a hybrid of traditional software
  72. development and systems administration. On paper this should be great. The silos
  73. will shrink. People will understand the limits and needs of the others. Managers
  74. will be able to have more flexible employees.</p>
  75. <p>Unfortunately though, a lot of the ideas behind devops and the overall
  76. philosophy really do require you to radically burn down everything and start
  77. from scratch. This tends to really not be conducive to engineering timetables
  78. and overall system stability during the age of turbulence.</p>
  79. <div class="conversation">
  80. <p class="conversation-chat">&lt;<b>Numa</b>&gt; What's the problem with burning everything down? Fire cleanses all things and
  81. purifies away the unworthy!</p>
  82. </div>
  83. <div class="conversation">
  84. <p class="conversation-chat">&lt;<b>Cadey</b>&gt; Not when you're the one being burned!</p>
  85. </div>
  86. <div class="conversation">
  87. <p class="conversation-chat">&lt;<b>Mara</b>&gt; Wait, so what actually happens then? Does it just end up being a sysadmin team
  88. made up out of coders?</p>
  89. </div>
  90. <p>Yeah, in practice this ends up being a "new team" or a reboot of an existing
  91. team in ways that is suddenly compelling or sexy to executives because a new
  92. buzzword is on the scene. Realistically, devops did end up getting a proper
  93. definition at a buzzword conference level (being able to handle development and
  94. deployment of services from editor to production), but in practice this ends up
  95. being just some random developers that you tricked into caring about production
  96. now while also telling them that they're better than the sysadmins.</p>
  97. <div class="conversation">
  98. <p class="conversation-chat">&lt;<b>Numa</b>&gt; Two jobs for the price of one!</p>
  99. </div>
  100. <p>This ends up shafting the sysadmin team even harder because the new fancy devops
  101. team has things they can talk about as positives for their quarters, so people
  102. can more easily make a case for promotion. As a sysadmin, your "success" case is
  103. "bad things didn't happen", which means success can't stand out on reviews.
  104. Consider "scaled production above the rate of our customer acquistion rate"
  105. against "set up continuous delivery to ensure velocity on our team, saving 50
  106. hours of effort per week". Which one of those do you think gets you promoted?
  107. Which one of those do you think gets headcount for new hires?</p>
  108. <p>This has human costs too. At one of my past jobs doing more sysadmin-y things
  109. (it was marketed as a devops hybrid role, but the "hybrid" part was more of
  110. "frantically patch up the sinking ship with code" and not traditional software
  111. development). Sleep is really essential to helping you function properly to do
  112. your job. During the times when I was pager bitch, there was at least a 1/8
  113. chance that I would be woken up in the middle of the night to handle a problem.
  114. I had to change my pager tone 15 times and still get goosebumps hearing those
  115. old sounds nearly a decade later. This ended up being a huge factor in my
  116. developing anxiety issues that I still feel today. I ended up getting addicted
  117. to weed really bad for a few years. I admit that I'm really not the most robust
  118. person in the world, but these things add up.</p>
  119. <div class="conversation">
  120. <p class="conversation-chat">&lt;<b>Cadey</b>&gt; I guess "addicted to weed" isn't totally accurate or inaccurate here, it's more
  121. that I was addicted to the feeling of being high rather than dependence on the
  122. drug itself. Either way, it was bad and weed was my cope. It also probably
  123. really didn't help that I was also starting hormone replacement therapy at the
  124. time, so I was going through second puberty at the time as well. This is the
  125. kind of human capital cost when dealing with dysfunction like this. I've always
  126. been kind of afraid to speak up about this.</p>
  127. </div>
  128. <p>However, there are real technical problems that can only really be solved from a
  129. devops perspective. Tools like Docker would probably never have happened in the
  130. way they did if the devops philosophy didn't exist.</p>
  131. <p><img src="https://cdn.christine.website/file/christine-static/blog/1BDBBB94-7052-4E4C-AE32-CFEE4226CBA8.jpeg" alt="A three panel meme with an old man talking to a child. The child says &quot;it works on my machine&quot;. The old man replies with &quot;then we'll ship your machine&quot;. The last panel says &quot;and that is how docker was born&quot;."></p>
  132. <p>In a way, Docker is one of the perfect examples of the devops philosophy. It
  133. allows developers to have their own custom versions of everything. They can use
  134. custom compilers that the sysadmins don't have to integrate into everything.
  135. They can experiment with new toolstacks, languages and build systems without
  136. worrying about how they integrate into existing processes. And in the process it
  137. defaults to things that are so hilariously unsafe that you only really realize
  138. the problems when they own you. It makes it easy to ship around configurations
  139. for services yes, but it doesn't make supply chain management easy at all.</p>
  140. <div class="conversation">
  141. <p class="conversation-chat">&lt;<b>Mara</b>&gt; Wait, what about that? How does that make any sense?</p>
  142. </div>
  143. <p>Okay, let's consider this basic Dockerfile that builds a Go service. If you
  144. start from very little knowledge of what's going on, you'd probably end up with
  145. something like this:</p>
  146. <pre><code class="language-Dockerfile">
  147. <span>FROM golang:1.17
  148. </span><span>
  149. </span><span>WORKDIR /usr/src/app
  150. </span><span>
  151. </span><span>COPY go.mod go.sum ./
  152. </span><span>RUN go mod download &amp;&amp; go mod verify
  153. </span><span>
  154. </span><span>COPY . .
  155. </span><span>RUN go build -v -o /usr/local/bin/app ./...
  156. </span><span>
  157. </span><span>CMD ["app"]
  158. </span>
  159. </code></pre>
  160. <p>This allows you to pin the versions of things like the Go compiler without
  161. bothering the sysadmin team to make it available, but in the process you also
  162. don't know what version of the compiler you are actually running. Let's say that
  163. you have all your Docker images built with CI and that CI has an image cache set
  164. up (as is the default in many CI systems). On your laptop you may end up getting
  165. the latest release of Go 1.17 (at the time of writing, this is version 1.17.8),
  166. but since CI may have seen this before and may have an old version of the <code>1.17</code>
  167. tag cached. This would mean that despite your efforts at making things easy to
  168. recreate, you've just accidentally put <a href="https://github.com/golang/go/issues/50165">an ASN.1 parsing
  169. DoS</a> into production, even though
  170. your local machine will never have this issue! Not to mention if the image
  171. you're using has a glibc bug, a DNS parsing bug or any issue with one of the
  172. packages that makes up the image.</p>
  173. <div class="conversation">
  174. <p class="conversation-chat">&lt;<b>Mara</b>&gt; So as a side effect of burning down everything and starting over you don't
  175. actually get a lot of the advantages that the old system had in spite of the
  176. dysfunction?</p>
  177. </div>
  178. <div class="conversation">
  179. <p class="conversation-chat">&lt;<b>Cadey</b>&gt; Yep! Realistically though you can get around this by using exact sha256 hashes
  180. of the precise Docker image you want, however this isn't the <em>default</em> behavior
  181. so nobody will really know about it. There are ways to work around this with
  182. tools like Nix, but that is a topic for another day.</p>
  183. </div>
  184. <p>This is what the devops experience feels like, chaining together tools that
  185. require careful handling to avoid accidental security flaws in ways that the
  186. traditional sysadmin team approach fundamentally avoided by design. By
  187. sidestepping the sysadmin team's stability and process, you learn nothing from
  188. what they were doing.</p>
  189. <div class="conversation">
  190. <p class="conversation-chat">&lt;<b>Cadey</b>&gt; This is all of course assuming that at the same time as you go devops, you also
  191. avow the grandeur of the cloud. Statistics say that these two usually go hand in
  192. hand as the cloud is sold to executives as good for
  193. devops.</p>
  194. </div>
  195. <p>As for how to get out of this mess though, I'm not sure. Like I said, this is a
  196. <em>social</em> problem that is trying to be solved through a <em>business organizational</em>
  197. fix. I am a technical solutions kind of person and as such I'm really not the
  198. right person to ask about all this. I don't want to propose a solution here.
  199. I've thought out several ideas, but I got nowhere with them fast.</p>
  200. <p>I remember at one of my jobs where I was a devops I ended up also having to be
  201. the tutor on how fundamental parts of the programming language they are using
  202. work. This one service that was handling a lot of production load had issues
  203. where it would just panic and die randomly when a very large customer was trying
  204. to view a list of things that was two orders of magnitude larger than other
  205. customers that use that service. I eventually ended up figuring out where the
  206. issue was but then I had an even harder time explaining what concurrency does at
  207. a fundamental level and how race conditions can make things crash due to
  208. undefined behavior. I think it ended up being a 3 line fix too.</p>
  209. <p>I guess the thing that would really help with this is education and helping
  210. people hone their skills as developers. I understand that there's a learning
  211. curve and not everyone is going to become a programming god overnight, but every
  212. little bit sets off butterfly effects that will ripple down in other ways. Any
  213. solution that requires everyone be a programming god isn't viable for anyone,
  214. including programming gods.</p>
  215. <div class="conversation">
  216. <p class="conversation-chat">&lt;<b>Numa</b>&gt; This whole mentorship thing only really works when the company you work for
  217. doesn't de-facto punish you for mentoring people like that. If you aren't
  218. careful about how you frame this, doing that could make it difficult for you to
  219. prove yourself come review time. "Helped other people do their jobs better"
  220. doesn't really look good for a promotion committee.</p>
  221. </div>
  222. <div class="conversation">
  223. <p class="conversation-chat">&lt;<b>Mara</b>&gt; Yeah but what are you supposed to do if that kind of mentorship is what really
  224. helps motivate you as a person and is what you really enjoy doing? I don't
  225. really see "mentor" as a job title on any postings.</p>
  226. </div>
  227. <div class="conversation">
  228. <p class="conversation-chat">&lt;<b>Numa</b>&gt; There's always getting tired of trying to change things from within and then
  229. writing things out on a publicly visible blog, building up a bunch of articles
  230. over time. Then you could use that body of work as a way to meme yourself into
  231. hiring pipelines thanks to people sharing your links on aggegators like the
  232. orange site. It'd probably help if you also got a reputation as a shitposter,
  233. usually when people are able to openly joke about something that signals that
  234. they are pretty damn experienced in it.</p>
  235. </div>
  236. <div class="conversation">
  237. <p class="conversation-chat">&lt;<b>Cadey</b>&gt; You're describing this blog aren't you.</p>
  238. </div>
  239. <p>Like I said though, this is hard. A lot of the problems are actually structural
  240. problems in how companies do the science part of computer science. Structural
  241. problems cannot be solved overnight. These things take time, effort and patience
  242. to truly figure out and in the process you will fail to invent a light bulb many
  243. times over. Devops is probably a necessary evil, but I really wish that
  244. situations weren't toxic enough in the first place to require that evil.</p>