Hands-on Tutorials
Reimagining History in Color with Pix2Pix as an Assistant
Creatives and AI collaborating: how a trained pix2pix model can help with colorization
Humans are inherently creative. New tools unlock the creativity lying dormant in an untapped wave of makers that find unexpected ways to showcase their artistic abilities. According to a Pfeiffer Consulting study, even experienced artists appreciate new tech as an enabler of creativity. The study found that 74% of the artistic process, from envisioning a work of art to its realization, consists of tedious steps that many artists wouldn’t mind automating. This is where some artists feel that machine learning can be of assistance. As an amateur artist myself with graduate training in machine learning, I’ve been exploring this possibility. In this article, I’ll describe how I trained a pix2pix model to be a helpful assistant for colorizing black and white photos. The model’s output provided a very helpful base, reducing the manual work I needed to do in Adobe Photoshop to colorize and restore historical portraits from the Smithsonian’s National Portrait Gallery. Although I cannot claim the colorizations to be historically accurate, I am quite happy with the results!
If you’re new to colorization, take a look at some of the fascinating work artists produce via these Reddit threads here and here. In addition to the manual process of painting layers of color onto an original black and white photograph, colorization artists do extensive research based on similar color images and historical accounts to add color as accurately as possible. It is a labor-intensive process, but the results produced are often stunning. For instance, Mario Unger spent 3000 hours restoring and colorizing a collection of historical photos, but just look at his amazing results. I find that we can relate a little better to history when it looks more like the world we live in.
Recently, the Smithsonian shared millions of assets as open data, encouraging the general public to dive in and create. Included in the Smithsonian Open Access are many historical portraits from the National Portrait Gallery. Wouldn’t it be neat to colorize these portraits? Since I don’t have the refined skills of a colorization artist, I decided to train a pix2pix model to help me.
Training pix2pix for colorization
The pix2pix model is a conditional Generative Adversarial Network (cGAN) for image-to-image translation. For colorization, the model is trained to translate a black and white (b&w) image into a color image. The model is comprised of 2 subcomponents: (1) a generator trained to add color to b&w images and (2) a discriminator which learns to distinguish between original color images and those created by the generator. A color image dataset is used for training data. Color is removed from each image to create a b&w and color image pair. The generator learns to colorize the b&w images so that they fool the discriminator into thinking they are originals while also reproducing the original color of the image as closely as possible. The discriminator learns to recognize when an image was colorized by the generator. Through this competing process, the model learns to improve its colorizations over many iterations. Since the authors of the paper released their code, many researchers and artists have used pix2pix in creative ways. Examples are shown in the community contributions section here.
Now with some background on the model itself, let’s imagine pix2pix is the apprentice of a colorization artist. For the apprentice to learn how to translate images from black and white to color, we must show it many examples — the more vibrant the better. I used a subset of the COCO dataset (35k images), filtering for only the most colorful images. After training on this dataset, pix2pix learned a bit about what the world looks like in color, but it still did not know how to colorize images of people well. Since the goal was to colorize portraits, its next bit of training needed to be focused on this task. I created a smaller, high quality dataset of images of people (2k images) using my personal photos and select photos from Unsplash. Then, because older, historical images are often blurrier than modern digital photos, I also added a random blur to the training images. Now, pix2pix will build on what it had already learned from COCO and further train to translate slightly blurred b&w photos of people to color images.
Model Results
After training, my pix2pix assistant learned to be very helpful. Using the Smithsonian portraits as my test data, pix2pix added color to the b&w images quite well although biases from my training data were apparent. As my training set was small, the model was more prone to overfitting — that is, the model learned patterns specific to the training data, and applied those specifics to unseen images incorrectly. For instance, many of my training images contained people wearing lipstick and, as a result, some historical figures were given lipstick in the output 😬 . One easy way to improve the model is to use a bigger, more diverse training set.
Overall, the results were certainly not perfect, but still provided a really nice base to start from. While I do not have the refined skills of a colorization artist, pix2pix helped me along enough to try my hand at colorization.
Finishing touches using Photoshop
Using the model outputs, it was very easy to add finishing touches with Photoshop.
Because the base colors were already in the image, I used the eye dropper tool with a “3x3 sample average” to pick up colors from specific pieces of an image. After picking up the color, I used the paint brush tool under the “Color” mode to paint over areas that the model missed or colored incorrectly. This brush will only adjust the color in the image without changing pixel intensity values (i.e., the highlights/shadows, etc. will not be disturbed). The specific settings I used are highlighted in the figure above. Then, to remove damage from the original photos such as scratches or pen marks, I used the standard brush setting to paint over it. I spent less than 5 minutes editing per image. Keep scrolling to see the final results!
Final Results
Thanks for reading, and I hope you enjoyed the final results. Models like pix2pix have the potential to make a task like colorization more accessible. Skilled colorization artists may also find such models helpful in reducing the tedious work required to arrive at a finished product. They can use the time saved to focus on updating colors for historical accuracy or artistic flair. Many thanks to the Smithsonian for releasing such an incredible dataset for researchers to explore history — reimagining history in color in this case 🎨
Food for thought — what else can machine learning paired with creatives in the loop produce? 💡
Image References (links to originals):
3. Helen Keller
5. Lil Green
