MSE 5760: Spring 2025 HW 6 (due 05/04/25)
Topic: Autoencoders (AE) and Variational Autoencoders (VAE)
Background:
In this final homework, you will build a deep autoencoder, a convolutional 
autoencoder and a denoising autoencoder to reconstruct images of an isotropic composite 
with different volume fractions of fibers distributed in the matrix. Five different volume 
fraction of fibers are represented in the dataset and these form five different class labels for 
the composites. After the initial practice with AEs and reconstruction of images using latent 
vectors, you will build a VAE to examine the same dataset. After training the VAE (as best 
as you can using the free colab resources to reproduces images), you will use it to generate 
new images by randomly sampling datapoints from the learned probability distribution of 
the data in latent space. Finally, you will build a conditional VAE to not only generate new 
images but generate them for arbitrary volume fractions of fibers in the composite.
The entire dataset containing 10,000 images of composites with five classes of 
volume fractions of fibers was built by Zequn He (currently a Ph.D. student in MEAM in 
Prof. Celia Reina’s group who helped put together this course in Summer 2022 by designing 
all the labs and homework sets). Each image in the dataset shows three fibers of different 
volumes with circular cross sections. Periodic boundary conditions were used to generate 
the images. Hence, in some images, the three fiber particles may appear broken up into
more than three pieces. The total cross sectional area of all the fibers in each image can, 
however, be divided equally among three fibers. Please do not use this dataset for other 
work or share it on data portals without prior permission from Zequn He
(hezequn@seas.upenn.edu).
Due to the large demands on memory and the intricacies of the AE-VAE 
architecture, the results obtained will not be of the same level of accuracy and quality that 
was possible in the previous homework sets. No train/test split is recommended as all 
10,000 images are used for training purposes. You may, however, carry out further analysis 
using train/test split or by tuning the hyperparameters or changing the architecture for 
bonus points. The maximum bonus points awarded for this homework will be 5.
**********************************Please Note****************************
Sample codes for building the AE, VAE and a conditional GAN were provided in 
Lab 6. There is no separate notebook provided for the homework and students will 
have to prepare one. Tensorflow and keras were used in Lab 6 and is recommended 
for this homework. You are welcome to use other libraries such as pytorch.
************************************************************************
1. Model 1: Deep Autoencoder model (20 points)
Import the needed libraries. Load the original dataset from canvas. Check the 
dimensions of each loaded image for consistency. Scale the images.
1.1 Print the class labels and the number of images in each class. Print the shape of 
the input tensor representing images and the shape of the vector representing the 
class labels. (2 points)
1.1. A measure equivalent to the volume fraction of fibers in each composite image is 
the mean pixel value of the image. As the images are of low-resolution, you may 
notice a slight discrepancy in the assigned class value of the image and the 
calculated mean pixel intensity. As the resolution of images increases, there will be 
negligible difference between the assigned class label and the pixel mean of the 
image. Henceforth, we shall use the pixel mean (PM) intensity of the images to be 
the class label. Print a representative sample of ten images showing the volume 
fraction of fibers in the composite along with the PM value of the image. (3 points)
1.2. Build the following deep AE using the latent dimension value = 64.
(a) Let the first layer of the encoder have 256 neurons.
(b) Let the second layer of the encoder have 128 neurons.
(c) Let the last layer of the encoder be the context or latent vector.
(d) Use ReLU for the activation function in all of the above layers.
(e) Build a deep decoder with its input being the context layer of the encoder.
(f) Build two more layers of the decoder with 128 and 256 neurons, respectively. 
These two layers can use the ReLU activation function.
(g) Build the final layer of the decoder such that its output is compatible with the 
reconstruction of the original input shape tensor. Use sigmoid activation for the 
final output layer of the decoder.
(h) Use ADAM as your optimizer and a standard learning rate. Let the loss be the 
mean square error loss. Compile the AE and train it for at least 50 epochs.
(10 points)
1.3. Print the summary of the encoder and decoder blocks showing the output shape of 
each layer along with the number of parameters that need to be trained. Monitor 
and print the lossfor each epoch. Plot the loss as a function of the epochs. (2 points)
1.4. Plot the first ten reconstructed images showing both the original and reconstructed 
images. (3 points)
2. Model 2: Convolutional Autoencoder model (20 points)
2.1 Build the following convolutional AE with the latent dimension = 64
(a) In the first convolution block of the encoder, use 8 filters with 3x3 kernels, 
ReLU activation and zero padding. Apply max pooling layer with a kernel of 
size 2.
(b) In the second convolution block use 16 filters with 3x3 kernels, ReLU activation 
and zero padding. Apply max pooling layer with a kernel of size 2.
(c) In the third layer of the encoder use 32 filters with 3x3 kernels, ReLU activation 
and zero padding. Apply max pooling layer with a kernel of size 2.
(d) Flatten the obtained feature map and then use a Dense layer with ReLU 
activation function to extract the latent variables.
(d) Build the decoder in the reverse order of the encoder filters with the latent 
output layer of the encoder serving as the input to the decoder part.
(e) Use ADAM as your optimizer and a standard learning rate. Let the loss be the 
mean square error loss. Compile the convolutional AE and train it for at least 
50 epochs.
(10 points)
2.2 Print the summary of the encoder and decoder blocks showing the output shape of 
each layer along with the number of parameters that need to be trained. Monitor 
and print the lossfor each epoch. Plot the loss as a function of the epochs. (5 points)
2.3 Plot the first ten reconstructed images showing both the original and reconstructed 
images. (5 points)
3. Model 3: Denoising convolutional Autoencoder model (15 points)
3.1 Add a Gaussian noise to each image. Choose a Gaussian with a mean of zero and a 
small standard deviation, typically ~ 0.2. Plot a sample of five original images with 
noise. (3 points)
3.2 Use the same convolutional autoencoder as in Problem 2 but with noisy images fed 
to the encoder. Train and display all the information as in 2.2 and 2.3.
(12 points)
4. Model 4: Variational Autoencoder model (25 points)
4.1 Set the latent dimension of the VAE be 64. Build a convolutional autoencoder with 
the following architecture. Set the first block to have 32 filters, 3x3 kernels with 
stride = 2 and zero padding.
4.2 Build the second block with 64 filters, 3x3 kernels, stride =2 and zero padding. Use 
ReLU in both blocks. Apply max pooling layer with kernel of size 2x2.
4.3 Build an appropriate output layer of the encoder that captures the latent space 
probability distribution.
4.4 Define the reparametrized mean and variance of this distribution.
4.5 Build the convolutional decoder in reverse order. Apply the same kernels, stride 
and padding as in the encoder above. Choose the output layer of the decoder and 
apply the appropriate activation function.
4.6 Compile and train the model. Monitor the reconstruction loss, Kullback-Liebler 
loss and the total loss. Plot all three quantities for 500 epochs. (10 points)
4.7 Plot the first ten reconstructed images along with their originals. (5 points)
4.8 Generate ten random latent variables from a standard Gaussian with mean zero and 
unit variance. Display the generated images from these random values of the latent 
vector. Comment on the quality of your results and how it may differ from the input 
images. Mention at least one improvement that can be implemented which may 
improve the results. (3+3+4=10 points)
5. Model 5: Conditional Variational Autoencoder model (20 points)
A conditional VAE differs from a VAE by allowing for an extra input 
variable to both the encoder and the decoder as shown below. The extra label could 
be a class label, ‘c’ for each image. This extra label will enable one to infer the 
conditional probability that describes the latent vector conditioned on the class label 
‘c’ of the input. In VAE, using the variational inference principle, one infers the 
Gaussian distribution (by learning its mean and variance) of the latent vector 
representing each input ‘x’. In conditional VAE, one infers the Gaussian 
conditional distribution of the latent vector conditioned on the extra input variable 
‘label’.
For the dataset used in this homework, there are two advantages of the 
conditional VAE compared to the VAE: (i) the conditional VAE provides a cheap
way to validate the model by comparing the pixel mean of the generated images 
with the conditional class label values (pixel mean) in latent space used to generate 
the images. (ii) The trained conditional VAE can be used to generate images of 
composites with arbitrary volume fraction of fibers with sufficient confidence once 
the validation is done satisfactorily.
A conditional VAE. (source: https://ijdykeman.github.io/ml/2016/12/21/cvae.html)
A good explanation of the conditional VAE in addition to the resource cited in the 
figure above is this: https://agustinus.kristia.de/techblog/2016/12/17/conditional vae/.
A conditional GAN (cGAN) toy problem was shown in Lab 6 where the volume 
fraction (replaced by pixel mean for cheaper model validation) was the design 
parameter, and thus, the condition input into the cGAN. In this question, you will 
build a conditional VAE for the same task of generating new images of composites 
as in Problem 4 by randomly choosing points in the latent space. Since each point 
in the latent space represents a conditional Gaussian distribution, it also has a class 
label. Therefore, it becomes possible to calculate the pixel mean of a generated 
image and compare it with the target ‘c’ value of the random point in latent space. 
It is recommended that students familiarize themselves with the code for providing 
the input to the cGAN with class labels and follow similar logic for building the 
conditional VAE. You may also seek help from the TA’s if necessary.
5.1 Create an array that contains both images and labels (the pixel mean of each image). 
Note the label here is the condition and it should be stored in the additional channel 
of each image.
5.2 Use the same structure, activation functions and optimizer as the one used to build 
the VAE in Problem 4. Print the summary of the encoder and decoder blocks 
showing the output shape of each layer along with the number of parameters that 
need to be trained. (5 points)
5.3 Train the cVAE for 500 epochs. Plot the reconstruction loss, Kullback-Liebler loss 
and the total loss. Plot the first ten reconstructed images along with their originals. 
Include values of the pixel mean for both sets of images. (5 points)
5.4 Generate 10 fake conditions (i.e., ten volume fractions represented as pixel means 
evenly spaced within the range 0.1 to 0.4 as used in Lab 6) for image generation. 
Print the shape of the generated latent variable. Print the target volume fraction (or 
pixel mean). Show the shape of the array that combines the latent variables and fake 
conditions. Print the shape of the generated image tensor. (2 points)
5.5 Plot the 10 generated images. For each image show the generated condition (the 
pixel mean of each image generated in 5.4) and the pixel mean calculated from the 
image itself. (3 points)
5.6 Compare the set of generated images from the conditional VAE with the ones 
obtained in Lab 6 using cGAN. Comment on their differences and analyze the 
possible causes for the differences. (5 points)

請加QQ：99515681  郵箱：99515681@qq.com   WX：codinghelp