Beyond Metaphors and Misinformation: Machine Learning in AEC

Tyler Kvochick

I recently gave a presentation at the AIASF Next conference titled, “This Building Does Not Exist (Yet).” The title is a reference to the website, where machine learning researchers have generated photo-realistic images of non-existent people by feeding neural networks massive datasets of real faces.

Figure from NVIDIA StyleGan

My presentation revolved around an interactive exercise in which the audience learned core principles of deep learning by live-coding in Python notebooks. A few people participated gamely while others observed and discussed with interest. Some of the other discussions around machine learning that I heard at the conference outside of my presentation reflected a general lack of familiarity with the realities of this technology and how it will affect the built environment.

I see a lot of excitement around machine learning, but the popular discourse is always in vague metaphorical terms, with a lot of misinformation or overselling of what machine learning is truly capable of. As an architect-turned-machine-learning-researcher, one of my areas of focus is to help people within AEC move beyond the jargon and bring a realistic, sophisticated understanding of this technology into their practice.

One assertion I’d like to put out there is that machine learning will fundamentally change design disciplines in the near future, and this is something we need to be prepared for.

There are two simple reasons for this. The first is that these are just software systems; there’s nothing magical about them, and that means it’ll be quite simple to subsume them into existing software platforms. If you have a Google account, you may have noticed that Gmail has started to do predictive texting while writing emails, and that it has recently gotten much better than past solutions. This capability is based on neural networks (specifically, recurrent neural networks). Facebook famously has said they use seven different neural networks to analyze every image you upload. This is already happening in the organizations that have all the resources and knowledge around machine learning. We’re already seeing it be subsumed into software platforms, and I think we’ll see that in the design disciplines as well.

But the second reason is that it’s a very different kind of software. It’s a kind of software that’s amenable to how designers work. It’s an idea that can be called “programming by example.” The overwhelming lesson from the limits of machine learning research is that we can use arbitrarily deep models that have a simple structure and train them to do various similar tasks. These tasks can be “programmed” by just pointing them at various datasets, different collections of stuff, and this changes their behavior without changing the structure fundamentally. This is a significant shift from typical software development, where changing the application requires changing the structure of the code.

The quintessential example is the classifier, where you’re able to take the dataset of, for example, cats and not-cats, point the machine learning model at it, run an image through several stages that we call layers, and end up with a classification of whether or not it’s a cat. And over time this thing is able to learn to predict whether or not a new image is a cat. Slightly more sophisticated versions of this are now out-performing humans at very specific image recognition tasks. But these are state-of-the-art, world-class programs in their particular application area.


Closer to design and engineering is a process of generation in which you can reuse that classifier architecture as part of a two-neural-network setup where it’s paired with a generator that generates candidate images. So the classifier here is simply reading alternately from some dataset of images that you’re interested in, perhaps of architectural plans, and then you have a generator that’s generating candidate images. Simply by receiving an error signal on whether or not it’s generating a good image, the generator is learning to imitate the features in the dataset that you’ve shown it.

Generative adversarial network (GAN).

This network has only been trained on a small dataset so it wasn’t brilliant, but what it produced was at least evocative; it was getting pretty close at realizing that we have these networks of rooms organized into generally rectilinear shapes. Sometimes we have slightly more interesting buildings that have different profiles; there are different rooms inside of them. We can have hallways and stairs features, and all of this is happening with no a priori knowledge of what a drawing represents. There’s no knowledge of stairs or walls or wall construction or anything like this. The network is simply looking at pixel values and learning to regenerate the features that exist in architectural drawings.

I did an experiment several months ago where we took examples of past projects that we’ve completed for various buildings and turned that into a dataset to train a machine learning model to do some of the rote placement of technology infrastructure symbols on plans. This is something we spend a lot of time on in our business at TEECOM. We took some blank floor plans and paired those with the ultimate location of where technology infrastructure ended up in the final project that we produced. It’s a kind of zero-knowledge approach where the ultimate locations of these systems can be generated by a machine simply by conditional probabilities from reading architectural drawings.

Outlet placement

So the second assertion I’d like to put out there is that we’ll move from the explicit placement of individual objects and symbols on drawings and models to designers being able to guide generative models simply by showing the model different collections of stuff that has the qualities that they care about.

Every architectural office has a model library, a library of references, and various precedents that they look at. These things will become active elements of design as opposed to just static references that live in the background. I think we will ultimately see a kind of fading out of placing individual architectural elements by hand.

If we look back at the history of design technologies, we started drawing by hand, where we were responsible for placing all of the droplets of ink or pieces of graphite on the page in the necessary place to build something. Then we developed CAD systems where we had measuring built in by default and data redundancy, and it made it easier to do that process. Then we developed Building Information Modeling, where we can simply click two endpoints and there’s a library of elements that brings with it all the geometry necessary to represent this wall that we’re going to build. And on top of that, we had this idea that we called parametric design where those same elements are able to change their behavior and the way they look based on how they’re placed in the model.

So if we project this trend into the future, I think the next step is something like being able to steer a generative model with a dataset of qualities rather than having to place explicit elements by hand. We continue this trend of more flexible editing where the machine is capable of resolving which technical details are necessary for the effect that we’re trying to create.

In the generating plans experiment, we used a tiny amount of data and freely available computing tools, and we began to get into the realm of believable drawings. These things clearly aren’t commercial tools that are ready for production, but I believe that we’re sort of at the beginning of them being available to designers to impact architectural practice. I think that this is something that architects need to be at least conversant in, in the same way that we need to be conversant in mechanical engineering and structural engineering. I think it’s a trend that we need to take head-on.

Join the conversation

I’ve been working on a project called Architecture ex Machina, where I’ve been saving learning resources and writing some blog posts about how I think this will change the AEC industry. So if you’re interested, please join the conversation!