A place to cache linked articles (think custom and personal wayback machine)
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

пре 3 година
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371
  1. title: How to Build Good Software
  2. url: https://www.csc.gov.sg/articles/how-to-build-good-software
  3. hash_url: 6be8b8649cd695b922a43d15f74d4773
  4. <h2>Why Bad Software Happens to Good People</h2>
  5. <p>Bad software is one of the few things in the world you cannot solve with money. Billion dollar airlines have flight search apps that are often inferior to those built by groups of students. Established taxi
  6. companies the world over have terrible booking apps despite the threat they face from ride-sharing services. And painful corporate IT systems are usually projects with massive budgets, built over the course of many years. Whatever the cause of bad software is, it does not seem to be a lack of funding.</p>
  7. <p>Surprisingly, the root cause of bad software has less to do with specific engineering choices, and more to do with how development projects are managed. The worst software projects often proceed in a very particular way:</p>
  8. <p>The project owners start out wanting to build a specific solution and never
  9. explicitly identify the problem they are trying to solve. They then gather a long list of requirements from a large group of stakeholders. This list is then handed off to a correspondingly large external development team, who get to work building this highly customised piece of software from scratch. Once all the requirements are met, everyone celebrates as the system is launched and the project is declared complete.</p>
  10. <hr>
  11. <h5><em>
  12. The root cause of bad software has less to do with specific engineering choices, and more to do with how development projects are managed.
  13. </em></h5>
  14. <hr>
  15. <p> However, though the system technically meets specifications, severe issues are
  16. found when it is put in the hands of actual users. It is slow, confusing, and
  17. filled with subtle bugs that make using it an exercise in frustration. Unfortunately,
  18. by this time the external development team has been dismissed and there
  19. are no resources left over to make the necessary fixes. By the time a new project
  20. can be initiated years later, all knowledge of what caused these problems has left
  21. the organisation and the cycle starts over again.</p>
  22. <p></p>
  23. <p class="small-text">A Conversation with Li Hongyi (Part 1)<br>
  24. <em>Scroll down for more videos</em></p>
  25. <p>The right coding language, system architecture, or interface design will
  26. vary wildly from project to project. But there are characteristics particular
  27. to software that consistently cause traditional management practices to fail, while allowing small startups to succeed with a shoestring budget:</p>
  28. <p>• Reusing good software is easy;
  29. it is what allows you to build good
  30. things quickly;<br>
  31. • Software is limited not by the amount
  32. of resources put into building it, but
  33. by how complex it can get before it
  34. breaks down; and<br>
  35. • The main value in software is not the
  36. code produced, but the knowledge
  37. accumulated by the people who
  38. produced it.</p>
  39. <p>
  40. </p>
  41. <p>Understanding these characteristics may not guarantee good outcomes, but it
  42. does help clarify why so many projects produce bad outcomes. Furthermore,
  43. these lead to some core operating principles that can dramatically improve
  44. the chances of success:</p>
  45. <p>
  46. </p>
  47. <p>1. Start as simple as possible;<br>
  48. 2. Seek out problems and iterate; and<br>
  49. 3. Hire the best engineers you can.</p>
  50. <p>While there are many subtler factors to consider, these principles form a foundation that lets you get started building good software.</p>
  51. <h2>Reusing Software Lets You Build Good Things Quickly</h2>
  52. <p>Software is easy to copy. At a mechanical level, lines of code can literally be copied and pasted onto another computer. More generally, the internet is full of tutorials on how to build different kinds of systems using ready-made code modules that are available online. Modern software is almost never developed from scratch. Even the most innovative applications are built using existing software that has been combined and modified to achieve a new result.</p>
  53. <p>The biggest source of reusable code modules is the open source community. Open source software is software in which code is freely published for anyone to see and use. Many of the largest contributors to the open source community are giant tech companies. If you want to use a state-of-the-art planet scalable database as Facebook does, just download the code for Cassandra that they open sourced in 2008. If you want to try out Google’s cutting-edge machine learning for yourself, download the TensorFlow system published in 2015. Using open source code does not just make your application development faster, it gives you access to technology that is far more sophisticated than anything
  54. you could have developed yourself. For the most popular open source code, it is even more secure as there are many more people paying attention and fixing vulnerabilities. This is the reason digital technology has made such rapid progress: even the newest engineers can build upon the most advanced tools our profession has to offer.</p>
  55. <p>The advent of cloud services has taken reusability even further, offering the full use of even proprietary systems for just a subscription fee. Need a simple website? Just configure one in a few clicks using a website building service like Squarespace or Wix. A database? Subscribe to a virtual one from Amazon Web Services or Microsoft Azure. Cloud services allow developers to benefit from specialisation; the service provider handles the setup, maintenance, and continued development of a reliable, high-quality piece of software that is used by all its subscribers. This allows software developers to stop wasting time on solved problems and instead focus on delivering actual value.</p>
  56. <p>You cannot make technological progress if all your time is spent on rebuilding existing technology. Software engineering is about building automated systems, and one of the first things that gets automated away is routine software engineering work. The point is to understand what the right systems to reuse are, how to customise them to fit your unique requirements, and fixing novel problems
  57. discovered along the way.</p>
  58. <hr>
  59. <h5><em>
  60. Software engineering is about building automated systems, and one of the first things that gets automated away is routine software engineering work.
  61. </em></h5>
  62. <hr>
  63. <h2>Software Is Limited by Complexity</h2>
  64. <p>How useful a piece of software can be is usually limited by its complexity rather than the amount of resources invested in building it.</p>
  65. <p>IT systems are often full of features but are still hated by users because of how confusing they become. In contrast, highly ranked mobile apps tend to be lauded for their simplicity and
  66. intuitiveness. Learning to use software is hard. Beyond a point, new features actually make things worse for users because the accumulated complexity starts to become overwhelming. For example, after serving as the hub of Apple’s media ecosystem for almost 20 years, iTunes was split into three
  67. different apps (for music, podcasts, and TV shows) this year, as its features had grown too complex for one app to handle. From a usability perspective, the limit is not how many features can be implemented, but rather what can fit into a simple intuitive interface.</p>
  68. <p>Even ignoring usability, engineering progress slows to a halt once a project becomes too complex. Each new line of code added to an application has a chance of interacting with every other
  69. line. The bigger an application’s codebase, the more bugs are introduced whenever a new feature is built. Eventually, the rate of work created from new bugs cancels out the rate of work done from feature development. This is known as “technical debt” and is the main challenge in professional software development. It is the reason why many large IT systems have issues that go unfixed for years.
  70. Adding more engineers to the project just adds to the chaos: they start running
  71. faster in place as the codebase keels over from its own weight.</p>
  72. <hr>
  73. <h5><em>
  74. Building good software involves alternating cycles of expanding and reducing complexity.
  75. </em></h5>
  76. <hr>
  77. <p>In such cases, the only way forward is to take a step back to rationalise and
  78. simplify the codebase. The system architecture can be redesigned to limit
  79. unexpected interactions. Non-critical features can be removed even if they
  80. have already been built. Automated tools can be deployed to check for bugs and
  81. badly written code. Bill Gates once said “Measuring programming progress by
  82. lines of code is like measuring aircraft building progress by weight”. Human minds can only handle a finite amount of complexity, so how sophisticated a software system can get depends on
  83. how efficiently this complexity budget is used.</p>
  84. <p>Building good software involves alternating cycles of expanding and reducing complexity. As new features are developed, disorder naturally accumulates in the system. When this messiness
  85. starts to cause problems, progress is suspended to spend time cleaning up. This two-step process is necessary because there is no such thing as platonically good engineering: it depends on your
  86. needs and the practical problems you encounter. Even a simple user interface such as Google’s search bar contains a massive amount of complexity under the surface that cannot be perfected in a single iteration. The challenge is managing this cycle, letting it get messy enough to make meaningful progress,
  87. but not letting it get so complicated that it becomes overwhelming.</p>
  88. <hr>
  89. <h5><em>
  90. There is no such thing as platonically good engineering: it depends on your needs and the
  91. practical problems you encounter.
  92. </em></h5>
  93. <hr>
  94. <h2>Software Is about Developing Knowledge More than Writing Code</h2>
  95. <p>In software development, most ideas are bad; this is not anyone’s fault. It is just
  96. that the number of possible ideas is so large that any particular idea is probably
  97. not going to work, even if it was chosen very carefully and intelligently. To make
  98. progress, you need to start with a bunch of bad ideas, discard the worst, and
  99. evolve the most promising ones. Apple, a paragon of visionary design, goes
  100. through dozens of prototypes before landing on a final product. The final
  101. product may be deceptively simple; it is the intricate knowledge of why this
  102. particular solution was chosen over its alternatives that allows it to be good.</p>
  103. <p>This knowledge continues to be important even after the product is built. If a new
  104. team takes over the code for an unfamiliar piece of software, the software will soon
  105. start to degrade. Operating systems will update, business requirements
  106. will change, and security problems will be discovered that need to be fixed. Handling these subtle errors is often harder than building the software in the first place, since it requires intimate
  107. knowledge of the system’s architecture and design principles.</p>
  108. <p></p>
  109. <p class="small-text">A Conversation with Li Hongyi (Part 2)<br>
  110. <em>Scroll down for more videos</em></p>
  111. <p>In the short term, an unfamiliar development team can address these problems with
  112. stopgap fixes. Over time though, new bugs accumulate due to the makeshift nature
  113. of the additional code. User interfaces become confusing due to mismatched
  114. design paradigms, and system complexity increases as a whole. Software should be
  115. treated not as a static product, but as a living manifestation of the development
  116. team’s collective understanding.</p>
  117. <hr>
  118. <h5><em>
  119. Software should be treated not as a static product, but as a
  120. living manifestation of the development team’s collective understanding.
  121. </em></h5>
  122. <hr>
  123. <p>This is why relying on external vendors for your core software development is
  124. difficult. You may get a running system and its code, but the invaluable knowledge
  125. of how it is built and what design choices were made leaves your organisation.
  126. This is also why handing a system over to new vendors for “maintenance” often
  127. causes problems. Even if the system is very well documented, some knowledge
  128. is lost every time a new team takes over. Over the years, the system becomes a
  129. patchwork of code from many different authors. It becomes harder and harder
  130. to keep running; eventually, there is no one left who truly understands how
  131. it works.</p>
  132. <p>For your software to keep working well in the long term, it is important to
  133. have your staff learning alongside the external help to retain critical engineering
  134. knowledge in your organisation.</p>
  135. <h2>3 Principles for Good Software Development</h2>
  136. <h4>1. Start as Simple as Possible</h4>
  137. <p>Projects that set out to be a “one-stop
  138. shop” for a particular domain are often
  139. doomed. The reasoning seems sensible
  140. enough: What better way to ensure your
  141. app solves people’s problems than by
  142. having it address as many as possible?
  143. After all, this works for physical stores
  144. such as supermarkets. The difference is
  145. that while it is relatively easy to add a
  146. new item for sale once a physical store
  147. is set up, an app with twice as many
  148. features is more than twice as hard to
  149. build and much harder to use.</p>
  150. <p>Building good software requires focus:
  151. starting with the simplest solution that
  152. could solve the problem. A well-made
  153. but simplistic app never has problems
  154. adding necessary features. But a big IT
  155. system that does a lot of things poorly
  156. is usually impossible to simplify and
  157. fix. Even successful “do it all” apps like
  158. WeChat, Grab, and Facebook started
  159. out with very specific functionality and
  160. only expanded after they had secured
  161. their place. Software projects rarely
  162. fail because they are too small; they fail
  163. because they get too big.</p>
  164. <hr>
  165. <h5><em>
  166. Software projects rarely fail because they
  167. are too small; they fail because they get too big.
  168. </em></h5>
  169. <hr>
  170. <p>Unfortunately, keeping a project focused
  171. is very hard in practice: just gathering
  172. the requirements from all stakeholders
  173. already creates a huge list of features.</p>
  174. <p></p>
  175. <p class="small-text">A Conversation with Li Hongyi (Part 3)<br>
  176. <em>Scroll down for last video</em></p>
  177. <p>One way to manage this bloat is by
  178. using a priority list. Requirements are
  179. all still gathered, but each are tagged
  180. according to whether they are absolutely
  181. critical features, high-value additions,
  182. or nice-to-haves. This creates a much
  183. lower-tension planning process because
  184. features no longer need to be explicitly
  185. excluded. Stakeholders can then more
  186. sanely discuss which features are the
  187. most important, without worrying about
  188. something being left out of the project.
  189. This approach also makes explicit the
  190. trade-offs of having more features.
  191. Stakeholders who want to increase
  192. the priority for a feature have to also
  193. consider what features they are willing
  194. to deprioritise. Teams can start on the
  195. most critical objectives, working their way
  196. down the list as time and resources allow.</p>
  197. <p>We followed a similar process for all
  198. our most successful apps. Form.gov.sg
  199. started out as a manual Outlook Macro
  200. that took us six hours to set up for
  201. our first user but today has processed
  202. about a million public submissions.
  203. Data.gov.sg started out as a direct copy
  204. of an open source project and has since
  205. grown to over 300,000 monthly visits.
  206. Parking.sg had a massive list of almost
  207. 200 possible features that we never
  208. got around to building but still has over
  209. 1.1 million users today. These systems
  210. are well received not in spite of their
  211. simplicity but because of it.</p>
  212. <h4>2. Seek Out Problems and Iterate</h4>
  213. <p>In truth, modern software is so complicated and changes so rapidly that no amount of
  214. planning will eliminate all shortcomings. Like writing a good paper, awkward
  215. early drafts are necessary to get a feel of what the final paper should be. To
  216. build good software, you need to first build bad software, then actively seek out problems to improve on your solution.</p>
  217. <p>This starts with something as simple as talking to the actual people you are trying to help. The goal is to understand the root problem you want to solve and avoid jumping to a solution based just on preconceived biases. When we first started on Parking.sg, our hypothesis was that enforcement officers found it frustrating to have to keep doing the mental calculations regarding paper coupons. However, after spending just one afternoon with an experienced officer, we discovered that
  218. doing these calculations was actually quite simple for someone doing it professionally. That single conversation saved us months of potentially wasted effort and let us refocus our project on helping drivers instead. </p>
  219. <p>Beware of bureaucratic goals masquerading as problem statements. “Drivers feel frustrated when dealing with parking coupons” is a problem. “We need to build an app for drivers as part of our Ministry Family Digitisation Plans” is not. “Users are annoyed at how hard it is to find information on government websites” is a problem. “As part of the Digital Government Blueprint, we need to rebuild our websites to conform to the new design service standards” is not. If our end goal is to make citizens’ lives better, we need to explicitly acknowledge the things that are making their lives worse.</p>
  220. <p>Having a clear problem statement lets you experimentally test the viability of different solutions that are too hard to determine theoretically. Talking to a chatbot may not be any easier than navigating a website, and users may not want to install yet another app on their phones no matter how secure it makes the country. With software, apparently obvious solutions often have fatal flaws that do not show up until they are put to use. The aim is not yet to build the final product, but to first identify these problems as quickly and as cheaply as possible. Non-functional mock-ups to test interface designs. Semi-functional mock-ups to try different features. Prototype code, written hastily, could help garner feedback more quickly. Anything created at this stage should be treated as disposable. The desired output
  221. of this process is not the code written, but a clearer understanding of what the right thing to build is.</p>
  222. <hr>
  223. <h5><em>
  224. Beware of bureaucratic goals masquerading as problem statements. If our end goal is to make citizens’ lives better, we need to explicitly acknowledge the things that are making their lives worse.
  225. </em></h5>
  226. <hr>
  227. <p>With a good understanding of the right solution, you can start work on building the actual product. You stop exploring new ideas and narrow down to identifying problems with your particular implementation. Begin with a small number of testers who will quickly spot the obvious bugs that need to be fixed. As problems are addressed, you can increasingly open up to a larger pool who will find more esoteric issues.</p>
  228. <p>Most people only give feedback once. If you start by launching to a large audience, everyone will give you the same obvious feedback and you’ll have nowhere to go from there. Even the best product ideas built by the best engineers will start out with significant issues. The aim is to repeatedly refine the output,
  229. sanding down rough edges until a good product emerges.</p>
  230. <p>Even after all this iteration, after launch is when problems with a product matter
  231. the most. A problem that happens only 0.1% of the time may not get noticed
  232. during testing. But once you have a million users, every day the problem
  233. goes unresolved is a thousand more angry people you have to deal with. You need to fix problems caused by new mobile devices, network outages, or security attacks before they cause
  234. substantial harm to your users. With Parking.sg we built a series of secondary
  235. systems that continuously check the main system for any discrepancies in
  236. payments, duplicate parking sessions, and application crashes. Building up
  237. an “immune system” over time lets you avoid being overwhelmed as new issues
  238. inevitably come up.</p>
  239. <p>Overall, the approach is to use these different feedback loops to efficiently
  240. identify problems. Small feedback loops allow for quick and easy correction
  241. but miss out on broader issues. Large feedback loops catch broader issues but
  242. are slow and expensive. You want to use both, resolving as much as possible with
  243. tight loops while still having wide loops to catch unexpected errors. Building
  244. software is not about avoiding failure; it is about strategically failing as fast
  245. as possible to get the information you need to build something good.</p>
  246. <h4>3. Hire the Best Engineers You Can</h4>
  247. <p>The key to having good engineering is having good engineers. Google,
  248. Facebook, Amazon, Netflix, and Microsoft all run a dizzying number of the largest
  249. technology systems in the world, yet, they famously have some of the most
  250. selective interview processes while still competing fiercely to recruit the strongest
  251. candidates. There is a reason that the salaries for even fresh graduates have
  252. gone up so much as these companies have grown, and it is not because they
  253. enjoy giving away money.</p>
  254. <p>Both Steve Jobs and Mark Zuckerberg have said that the best engineers are at
  255. least 10 times more productive than an average engineer. This is not because
  256. good engineers write code 10 times faster. It is because they make better
  257. decisions that save 10 times the work.</p>
  258. <p>A good engineer has a better grasp of
  259. existing software they can reuse, thus
  260. minimising the parts of the system they
  261. have to build from scratch. They have
  262. a better grasp of engineering tools,
  263. automating away most of the routine
  264. aspects of their own job. Automation
  265. also means freeing up humans to work
  266. on solving unexpected errors, which the
  267. best engineers are disproportionately
  268. better at. Good engineers themselves
  269. design systems that are more robust
  270. and easier to understand by others.
  271. This has a multiplier effect, letting their
  272. colleagues build upon their work much
  273. more quickly and reliably. Overall, good
  274. engineers are so much more effective
  275. not because they produce a lot more
  276. code, but because the decisions they
  277. make save you from work you did not
  278. know could be avoided.</p>
  279. <p></p>
  280. <p class="small-text">A Conversation with Li Hongyi (Part 4)</p>
  281. <p>This also means that small teams of the
  282. best engineers can often build things
  283. faster than even very large teams of
  284. average engineers. They make good
  285. use of available open source code and
  286. sophisticated cloud services, and offload
  287. mundane tasks onto automated testing
  288. and other tools, so they can focus on the creative problem-solving aspects of
  289. the job. They rapidly test different ideas
  290. with users by prioritising key features
  291. and cutting out unimportant work. This
  292. is the central thesis of the classic book
  293. “The <em>Mythical Man-Month</em>”<a href="#notes"><sup>1</sup></a>: in general,
  294. adding more software engineers does
  295. not make a project go faster, it only
  296. makes it grow bigger.</p>
  297. <hr>
  298. <h5><em>
  299. Building software is not about avoiding failure; it is about strategically failing as fast as possible to get the information you need to build something good.
  300. </em></h5>
  301. <hr>
  302. <p>Smaller teams of good engineers will
  303. also create fewer bugs and security
  304. problems than larger teams of average
  305. engineers. Similar to writing an essay, the
  306. more authors there are, the more coding
  307. styles, assumptions, and quirks there
  308. are to reconcile in the final composite
  309. product, exposing a greater surface area
  310. for potential issues to arise. In contrast,
  311. a system built by a smaller team of
  312. good engineers will be more concise,
  313. coherent, and better understood by
  314. its creators. You cannot have security
  315. without simplicity, and simplicity is rarely
  316. the result of large-scale collaborations.</p>
  317. <p>The more collaborative an engineering
  318. effort, the better the engineers need
  319. to be. Problems in an engineer’s code
  320. affect not just his work but that of his
  321. colleagues as well. In large projects,
  322. bad engineers end up creating more
  323. work for one another, as errors and
  324. poor design choices snowball to create
  325. massive issues. Big projects need to
  326. be built on solid reliable code modules
  327. in an efficient design with very clear
  328. assumptions laid out. The better your
  329. engineers, the bigger your system can
  330. get before it collapses under its own
  331. weight. This is why the most successful
  332. tech companies insist on the best talent
  333. despite their massive size. The hard limit
  334. to system complexity is not the quantity
  335. of engineering effort, but its quality.</p>
  336. <h2>Conclusion</h2>
  337. <p>Good software development starts
  338. with building a clear understanding of
  339. the problem you want to solve. This
  340. lets you test many possible solutions
  341. and converge on a good approach.
  342. Development is accelerated by reusing
  343. the right open source code and cloud
  344. services, granting immediate access
  345. to established software systems and
  346. sophisticated new technology. The
  347. development cycle alternates between
  348. exploration and consolidation, quickly
  349. and messily progressing on new ideas,
  350. then focusing and simplifying to keep
  351. the complexity manageable. As the
  352. project moves forward, it gets tested
  353. with successively larger groups of people
  354. to eliminate increasingly uncommon
  355. problems. Launching is when the real work
  356. ramps up for a good development team:
  357. layers of automated systems should be
  358. built to handle issues quickly and prevent
  359. harm to actual users. Ultimately, while
  360. there are infinite intricacies to software
  361. development, understanding this process
  362. provides a basis to tackle the complexities
  363. of how to build good software.</p>