This post is mirrored on my blog.
I started my morning off with Alan Kay's talk Programming and
Scaling. There are several
things I agree with and disagree with in that talk. It got me thinking
about software as a whole, and my part to play in it.
Background
I have long been a fan of Jonathan Blow and Casey Muratori. Two talks in
particular have been especially influential for me:
My other programming hero, John Carmack, has a different opinion of
software, where things are generally pretty good. From his Twitter:
Just last night I finished a book (How to Take Over the World:
Practical Schemes and Scientific Solutions for the Aspiring
Supervillain) with a related data point that made me a bit sad. I
remain a strident optimist, and there is a ton of objective data to
back that position, but I do feel that broad pessimism is an
unfortunate brake on progress.
Fiery indignation about a flaw, when directly engaged in building a
solution is a fine path to progress, but "pessimism cheerleading"
and self-flagellation have few upsides. Being an optimist, I feel that
progress is going to win anyway, but I would rather win faster!
I can't find a direct quote about software, but I believe he would
agree that he is optimistic about software as well. I have a hard time
reconciling this, because he has created really amazing things and knows
way more than I do about programming, but I still strongly feel we are
not on the right track as an industry.
Conway's law
Casey Muratori's talk The Only Unbreakable
Law is a must-watch. He
talks about Conway's law, which is basically that organizations can
only create things which replicate the organizational structure.
This is something I've had personal experience fighting in the game
industry. Game studios frequently organize based on roles, which means
there is a design team, a gameplay programming team, an art team, etc.
Communication is naturally more expensive between teams, which leads to
team power struggles, out-of-sync feature development, poor product
coherency, and other issues.
For example, Design requests a feature which doesn't get implemented
for weeks until Gameplay gets around to it. If the designer instead sat
next to the gameplay engineer and brainstormed with them, from the very
inception of the idea the engineer has been involved. This gives the
engineer opportunities to suggest designs which uniquely leverage the
technology, and gives the designer opportunities to have a more
realistic picture of the feasibility of there ideas. (I have heard of
design throwing out ideas thinking that they would be too hard to
implement, but are actually technically simple, for example.) With the
two people sitting together, the chances are high that the feature will
in by the end of the day.
Methodologies like Scrum try to combat siloization by having
cross-disciplinary teams--e.g. a designer, engineer, and artist all
working closely together. Note, however, that you are still splitting a
project into separate teams. If you have a Quests team and a Combat
team, chances are lower that quests and combat will be as coherent with
each other than if you had simply a "gameplay" team with no artificial
divisions.
Open source development
The story gets even more complicated when you enter the free software
world. Now, there aren't even teams, but frequently individual
contributors, who both
- don't have as close an understanding as the project's original
developer
- likely have a lower-bandwidth communication with the project's
owners.
The FOSS world still naturally creates divisions here: there's the
trusted core development team, and everyone else. This is a natural
consequence of there being differing levels of quality, experience, and
understanding between contributors.
At a more macro scale, you have teams of contributors working on single
libraries or programs. There is rarely ever coherence between two
programs, for example GIMP and Emacs. There's no one leading the
overall system's direction. This has an advantage in the freedom it
provides, but the sacrifice is increased system complexity, redundant
work, and overall lower quality.
If you have N developers working on M different photo editors, you
likely will end with M photo editors at 80% completion; if you have them
all work on one editor, it seems you would end up with one 100% photo
editor. However, even as I say this I know it's a fallacy, because
having more people work on something does not imply higher quality--see
both Brooks' Mythical Man Month and Conway's law again. I think this
shows how natural that line of thinking is.
One could instead imagine the M different editors being different
attempts to make something great, and then let natural selection take
over. This seems like a reasonable away to approach innovation, but has
a few problems:
- People will make things without understanding the state of the art
and the previous attempts, causing them to waste time retracing
steps.
- People can copy-cat programs and make things with only trivial
innovations, causing programs to get stuck in local maxima. (Though
evolutionarily, this would be the equivalent to mutation, which can
end up a winning strategy in the very long term. Ideally, we would
be able to do things faster thanks to our ability to think and
react, but it's possible at the macro scale we can't beat out
evolution. I'm out of my depth here.)
- End-users likely don't care about there being ten different
innovative attempts at editing photos. They just want one program
that does everything they want. I don't believe this is a good
counter-argument, however. People have different ways of thinking
about and doing things, which necessitates different programs--there
will never be a program that can please everybody. In fact, the more
people you try to please, you either have to simplify the program
(to appeal to beginner users; this is largely the strategy SaaS
seems to take) or bloat the program with every feature you can think
of.
A system created by ten people sitting in a room together is guaranteed
to be more conceptually sound than one created by thousands. This
doesn't necessarily mean the ten-person program will end up better,
but the chances of all 10 people being on the same page are vastly
higher than all those thousands.
I do not want my argument to be used as one against free software; I
only want to point out that small, tight teams are likely to produce
more coherent programs. I don't think the proprietary vs. free applies
here, except that it seems easier to convince ten people to sit in the
same room for a few years if you're paying them.
Local maxima
Systems can grow to the point where making a necessary change takes so
long and is so costly that the change cannot happen without large
rework. In order to escape local maxima, we must be willing to start
over.
Over time, software seems to increase the amount of dependencies it has.
For example, an abstraction library like SDL will add code for X11, then
Wayland, then Win32, then Apple's Metal, etc. New technology is
invented, which is a good thing, but the question becomes, well, what do
we do with all the old stuff?
How Microsoft Lost the API
War
discusses exactly this with regard to Microsoft. He reaches the
conclusion that the Web is the ultimate solution, which I refuse to
accept, and which I believe end-users also dislike. We still install
"apps", and we still care about latency, even if we don't know it;
things are so bad it's easy to forget what good software feels like.
Maybe the existence of successful app markets is evidence that web
developers have done a really terrible job, that no UI standardization
severely limits app quality, or that the foundational web technology can
never result in as good of experiences for users. Hey, maybe all these
are true.
While I'm on Joel on Software, I have to reference his argument
against ever re-writing
code.
I disagree with this essay when applied to the industry at large. It
might make sense if you are a startup with a limited runway or a
for-profit company, but this attitude in regard to overall industry
health and technological innovation is extremely harmful. We must not
only be willing to rewrite things, but encourage it! We should be
telling every new graduate, "study the mistakes of the past, and build
the new future!", not "don't ever start from scratch, just keep
piling on".[^1] We need new ideas and we need to try new approaches if
we are ever going to evolve.
Software engineering vs. engineering engineering
In that Alan Kay talk he references the construction of the Empire State
Building. This got me thinking how programming is fundamentally
different from construction engineering. At the core it's about the
difficulty of changing the system: a building is immensely difficult to
change after it is finished, whereas software can be relatively trivial
to modify.
This is something we should embrace. It is miraculous that we have a
technology that can so easily be molded
into different shapes. However, it comes with some interesting
side-effects at many levels.
For example, organizations feel more comfortable taking risks with
software. If they can fix the software after it has shipped, they are
more likely to take a "more, buggy features" approach than a "fewer,
higher quality features" approach and fix the bugs after. This is one
reason why technical debt grows even before a piece of software is
released--you're already operating under the assumption that you'll
fix it "later", because it's relatively easy to fix it then.
The lean startup methodology brought continuous deployment, which meant
that software gets shipped without even being tested, because even if
the end-users have a bad experience, the change can come so fast that it
will be fixed the very next day. I'm not endorsing this as a good
practice, just indicating it's something unique to software.
Ideally, we embrace the ease of changing to allow users to customize the
software and to allow software to be ported easily to new platforms. We
should also try to be aware of the negative impact frequent change has
on user experiences when we ship software that might frustrate them,
ruin their data, or put them in danger.
Burn it all to the ground
Project Oberon is "a design for a
complete desktop computer system from scratch. Its simplicity and
clarity enables a single person to know and implement the whole system,
while still providing enough power to make it useful and usable in a
production environment."
Alan Kay and several others formed Viewpoints Research
Institute partially to answer a similar question:
If we were to start from scratch, how much code would it take to get us
what we have?
Another great paper to read is Rob Pike's "polemic" Systems Software
Research is Irrelevant.
Many programmers thinking of these issues fantasize about doing these
kinds of rewrite projects. It takes both confidence and,
pessimistically, arrogance to believe that you can re-invent the
software world better than those who came before. However, we should not
discourage people from doing this, for all the reasons I have stated
previously:
- The system could be much stronger with fewer minds designing it
(Conway's law)
- The system could escape the local maxima the previous system was
trapped in (Accretion of complexity)
Problems with starting over
The hardest pill to swallow is the upfront cost of catching up. This
cost is largely levied by the use of existing technology. Complexity
seeps in from every level:
- Users are frequently quirky and understand things in different ways.
This is the domain of user interface/user experience design.
- People disagree, but must come together and create standards of
communication to utilize the physical infrastructure optimally.
These standards bring with them significant complexity. They get
revised every so often, thereby obsoleting existing software and
requiring new work be done to support the new standards. Backwards
compatibility can be a way to reduce this impact, but it can also
reduce innovation and impose a large cost on implementing a
standard.
- Programmers want to save time by layering on abstractions, each of
which adds complexity and possibility of bugs.
- Computer hardware is complicated, frequently in order to offer
higher performance (which I believe is frequently worth it because
it enables new possibilities).
- The physics of the universe itself make things difficult. Computers
get hot, bits must be transferred between computers via some medium,
there's EM noise and physical vibration, etc.
At a nitty-gritty software level, one might try to mitigate some of
these complexities by re-using software. This is made more difficult by
the explosion of dependencies between software. In the worst case I've
seen, a single JavaScript file to rasterize a font to an image resulted
in downloading code from three different languages (Python, C, and
JavaScript), took over 3 minutes to download on my 20 Mebibyte/second
connection, and failed to run due to missing pip
dependencies!
Software is a disaster.
Please, don't let this discourage you from trying to start something
from scratch! We need people who are willing to do the hard work to
advance the field.
Virtualize everything
A common head-in-the-sand approach to attacking complexity is to create
your own little virtual machine, then program against it.
This brings along with it numerous problems:
- You have to implement that virtual machine, which means you still
have to deal with that complexity; it's not a way to actually
eliminate the complexity at its source.
- Performance always degrades. Always. At the core of the
counter-argument to this is the "sufficiently smart compiler" that
will magically make the cost of virtualization/high abstraction go
away. It's not here yet, and I wouldn't bank on it being here for
a while. JIT compilers bring significant complexity and runtime
cost.
- Hardware matters. The industry has largely been profiting off of
orders-of-magnitude improvements in hardware. Virtualizing away what
makes one machine special or faster than another is
counter-productive.
- It's another layer which has overhead, has to be learned, can
break, people must agree on, etc. It's a liability. It's more.
We want less, not more.
A similar approach is to have strict abstraction layers which separate
your code from the "other stuff". Interestingly, this approach suffers
the same ills predicted by Conway's law. If I'm writing a game in C
using SDL as my abstraction layer, I am less likely to do things that
the operating system would allow, but SDL does not. It's possible for
me to modify SDL to expose the functionality, but there's more friction
in doing so, which means I'm less likely to do so.
I'm not trying to rag on using libraries, because it seems like a much
more sustainable approach than using gigantic game engines or
frameworks. It's a danger we still need to be aware of in all of these
approaches.
What's the solution?
The only solution I can think of to these problems is strong culture,
community, and education.
We cannot focus on a single piece of technology that will somehow save
us--we have to focus on the people. Smart, aligned people can make new
technology and tackle new problems.
Culture
People need to care about these problems. If no one believes we even
have problems[^2], then we of course will never efficiently solve them.
We need to encourage innovation. When someone starts writing a new
operating system, programming language, game engine, etc., we should be
cheering them on.
If someone isn't yet experienced enough to practically undertake the
project, we can advise them as much while still encouraging them by
saying e.g. "I don't think you are ready to undertake that scale of
project, but you will get there in time if you continue learning,
especially by studying X, Y, and Z."
In comparison, if you ever say "it's stupid to create an X, we already
have Y", think really hard about what positive influence you actually
have. Think about your Y, and what came before it. Would the creators of
that Y have been told the same thing, for their era? Wouldn't you
rather have a new, potentially better X in ten years, rather than
aborting it at its inception?
Similarly, we need to encourage digging deeper. We should encourage web
developers to learn about cache lines and GPU architectures. Game devs
should know how the internet works. Don't encourage people to only
learn the minimum amount necessary to complete their next task. Learning
as little as possible is not a virtue. Take joy in exploring all the
fascinating innovations. Take note of all the cruft you find on the way
so you can learn what not to do.
Community
You need people willing to do work to fix these problems. Those people
likely need to talk to other people to get advice or increase their
understanding.
This is one of the shining traits of free software. Knowledge gained
from studying proprietary software is restricted to that company and the
people who bring knowledge from other companies. Clearly an open
industry is superior; artificial restriction of knowledge can be nothing
but harmful to innovation.
As a small PSA, please don't start your community on a closed platform
like Discord. These platforms are not good for long-term archival and
search. The data aren't available to the public. You pour more fuel on
a closed, proprietary future rather than supporting an open and free
discourse, of which we have
multiple
free and
open protocols to
choose from.
Education
New people are constantly being born, growing up, and entering the
field. We need systems to educate people on how to combat these
problems.
If you think universities instill any good software development
practice, well, you haven't interviewed many new grads. This isn't
necessarily the fault of universities. They rarely have a stated goal to
create good programmers, and instead focus on the science of the field.
There's definitely value to that; the danger is assuming that someone
with a CS degree knows anything about writing good software. Don't
believe me? Next time you read a CS paper, actually check out the code
in their GitHub link. Again, writing good code is not really a goal of
papers, nor the university.
Technical schools are similarly suspicious. Coding bootcamps frequently
hire their own graduates to be instructors, meaning the instructors
never gain real industry experience or write significant programs before
parroting what they were taught. It's effectively playing the game of
Telephone with programming knowledge.
Education in the game industry
The game industry has GDC, a conference where developers gather to share
insights gained in the field. However, talks frequently cover the "what
we did" rather than the "how we arrived here" nor "what was tried
but ended up failing". Most GDC talks are pay-walled in GDC Vault.
Talks frequently are thinly veiled recruiting advertisements rather than
honest attempts at sharing knowledge. They are often mile-high views
without revealing any deep technical details.
The game industry does not have an open source software practice. id
Software, while under John Carmack, is one of the only AAA game studios
to release the entirety of their game
code under free software licenses.
Since Carmack left, they no longer follow this practice. Indie
developers are no better on this front, despite the drastically lower
legal oversight that might cause a corporation more friction in
releasing their code.
University courses in games are becoming more common, but courses which
teach game tech fundamentals (as opposed to specifically Unity or
Unreal) are becoming increasingly rare. People are graduating with
degrees that certify they know Unity, not how game engines actually
work. These practices do not bode well for innovation in engine
technology in the long term.
Conclusion
I am consistently frustrated with my experience writing software. What
is a fundamentally simple task is often made incredibly hard to
implement due to accidental complexity and cruft. I feel powerless to
fix everything, because I have limited time and limited willpower.
However, I continue to learn and grow, and hope that I can positively
impact the field by contributing to and participating in these solutions
I've proposed. At the end of the day, the magic of software keeps me
going, and I hope I can somehow spread that magic and amplify it with my
work.
If you have thoughts, I encourage you to share them with me at
[email protected].
[^1]: For those feeling motivated to jump right in, I will temper this
advice by saying that you should really know the actual problems and
state of the art before trying to solve them. This can take a few
years of study and experience before you will actually be able to
know where the problems and the frontiers are.
[^2]: I have raged against the Unity- and Unreal-ification of the game
industry
time
and time
again. It seems the majority opinion at both the indie and AAA level
is that moving to the big U engines is a good, perfect thing with no
drawbacks. The short-sightedness of the engine commoditization is
the second biggest problem in the game industry in my opinion, the
first being the rising the cost per byte (Raph Koster - Industry
Lifecycles). These two
are intimately related--the verdict on whether moving to big-U's is
better rests on the answer to the following question: are our games
getting better (higher quality, and not just graphically), easier to
make, and faster to deliver? If the answer is not "yes" to all of
these, there's something lost here.