# Attention Guided Image-to-Image translation of Food Images Using GANs
### Authors: Petar Tonchev and Mariya Mladenova

This repository and the base of our project is taken from 
https://github.com/Ha0Tang/AttentionGAN 


**All experiments and results are well-documented in the [project report.](https://gitlab.com/deep-food-ub/food-gan/-/raw/master/Project%20Report%20-%20Attention%20Guided%20Image-to-Imagetranslation%20of%20Food%20Images%20Using%20GANs.pdf)**

You can see a quick overview of the project in the following presentation:

<a href="https://docs.google.com/presentation/d/e/2PACX-1vQB87w_pCeF4ZvKmM8ImPffYJBiFWRrnPHMkL9Ww64Fv3CaqanX4-vlneNsvVC1Xrb6oaZBcpB3wvdU/pub?start=true&loop=false&delayms=3000&slide=id.g6c52a2e8d8_0_177">
  <img src="/uploads/fd97b1cadfee8bdf6e3c879b224fd0b5/Screenshot_2020-12-08_at_16.46.51.png"  width="530" height="300">
</a>



## Abstract

Nowadays, with the help of the novel machine learning models, in particular
Generative Adversarial Networks, we are able to generate synthetic media, which
look absolutely realistic and at the same time authentic. Still, the food 
image-to-image translation remains a challenging problem that is very
unexplored. Due to the complexity of food images the state of the art results 
are noisy and slow in covergence. In our work, we explore how
adding attention to the image-to-image translation on food data can produce 
more photo-realistic synthetic images and speed up the convergence of the algorithm.
Furthermore, we present extensive analysis of GANs for food image synthesis 
sharing our insights on this problem.


## Installation

```sh
git clone https://gitlab.com/deep-food-ub/food-gan.git
cd food-gan

pip install -r requirements.txt
```

### Dependencies:

```sh
torch>=0.4.1
torchvision>=0.2.1
dominate>=2.3.1
visdom>=0.1.8.3
wandb>=0.8.29
pytorch_msssim              # only for the convegence metrics branch
scikit-image                # only for the convegence metrics branch

```


## Data Preparation

Download the Food101 dataset from [here](https://www.kaggle.com/dansbecker/food-101/home).

Split the data into the test and train set according to the files in the `meta` directory
After this, organize the data in the following way:

```sh
.
├── datasets
|    ├── pizza2pasta
|    │   ├── trainA              # training pizza images
|    │   ├── trainB              # training pasta images
|    │   ├── testA               # testing pizza images
|    │   └── testB               # testing pasta images
```

## Training a model

* To view the training results and loss plots, run `python -m visdom.server` and go to http://localhost:8097
* Put your Weights and Biases api access key as a parameter in the script below `--wandb_access_key 'put your wandb acces key here'`
* Train the algorithm using
```sh
 python train.py --dataroot ./datasets/pizza2pasta --name pizza2pasta --model attention_gan --dataset_mode unaligned --pool_size 50 --no_dropout --norm instance --lambda_A 10 --lambda_B 10 --lambda_identity 0.5  --batch_size 2 --niter 800 --niter_decay 0 --gpu_ids 0 --display_freq 700 --print_freq 500 
```

## Generating images using a trained model

Once you have trained the model you can use it to generate new images by referreing to it with the name you have used above in the `--name`` parameter.
To generate images simply run the following script:

```bash
python test.py --dataroot ./datasets/pizza2pasta --name pizza2pasta --model attention_gan --dataset_mode unaligned --norm instance  --phase test --load_size 256 --crop_size 256 --batch_size 1 --gpu_ids 0 --saveDisk --num_test 1000000000000
```

where `num_test` specifies the maximum amount of images to process.

## Feature-Branches ogranization
The repository is organized by putting each of our experiments in a separate branch:

* `master` -> contains the base AttentionGAN architecture with the addition of 
Weights and Biases (https://www.wandb.com) integration.
* `added-masks` -> __A-GAN 15__ (the extended AttentionGAN architecture with 15 content masks)
* `page-net-added-masks` -> PAGE-Net integration to the scheme I of AttentionGAN
* `page-net-added-masks-v2` ->  __A-GAN 15 PAGE-Net__ (PAGE-Net integration with $`\gamma`$ scaling)
* `page-net-attention-only` -> PAGE-Net is used as both foreground and background attention in the A-GAN 15 architecture
* `convergence-metrics` -> __A-GAN 15__ tracking the MS-SSIM across epochs
* `diverse-attention` -> Base AttentionGAN with additional loss function that aims to guide the attention masks to focus on different objects on the image






## Brief Overview of the project

In our work we successfully apply for the first time attention guided image-to-image translation
of food images. The main architecture we use is AttentionGAN which we extend by
adding additional content masks and their corresponding foreground attentions.
Moreover, we propose integrating the attention mechanism of the PAGE-Net algorithm 
in order to improve the quality of the generated images. We compare these two architectures
with the current state of the art architecture for food-to-food image translation - CycleGAN.

We train all three architectures for 800 epochs on the Food101 dataset.
We follow the default AttentionGAN parameter settings:
we scale the images to 256x256, doing horizontal flips and random crops for data
augmentation. The optimizer is Adam with β1 = 0.5 and β2 = 0.999 momentum terms.
The loss scaling parameters are set as following: λcycle = 10, λid = 0.5. 
Due to the limited GPU memory of 11 Gb we were forced to run AttentionGAN 
experiments with batch size of 2, rather than the default size of 4.


## Visual Results

### Pizza to Pasta Translation
![Result](./img/pasta-comp-agan15.png)

![Result](./img/pasta-comp-pagenet.png)

### Pasta to Pizza Translation
![Result](./img/pizza-comp-agan15.png)

## Quantitative Results

## IS and FID

The Inception Score and the Frechet Inception Distance are shown in the table below:


#### IS scores, higher is better
|            Method                           | Inception Score:  |
|:-------------------------------------------:|:-----|
| A-GAN 15                                    | 2.88 |
| A-GAN 15 PAGE-Net                           | 3.06 |
| **CycleGAN**                                | **3.10** |
| Real Training Set                           | 2.88 |
| Hold-out Test Set                           | 3.06 |



#### FID scores, lower is better


|            Method               |         FID:  Pizza to Pasata | FID:    Pasta to Pizza  |
|:-------------------------------:|:-----------------------------:|-------------------------|
| A-GAN 15                        | **66.83** | **62.23** |
| A-GAN 15 PAGE-Net           |112.43                 | 74.56      |
| CycleGAN          | 72.89                 |  74.49     |
| Hold-out Test Set          | 41.06                | 33.31 |

## How good are the images at fooling a well trained classifier?

The most accurate classifier trained on Food101, that is publicly available, is
based on [DenseNet-161](https://github.com/Escanor1996/food_ classifier_deployment)
and it has 93.26% top 1 accuracy and 99.01% top 5 accuracy on all 101 classes.
We use it to classify the output of A-GAN 15 and applied filtering for anomalies and failed attentions,
A-GAN 15 PAGE-Net and CycleGAN.


###### Accuracy on Generated Pasta Images
|     Method       | Top 1 Accuracy | Top 5 Accuracy | Confidence Pasta > Confidence Pizza |
|:----------------:|:--------------:|:--------------:|:-----------------------------------:|
| __A-GAN 15__     | __62.7%__ | __84.9%__ | __79.9%__|
| A-GAN 15 PAGE-Net| 42.9% | 68.7% | 64.0% |
| CycleGAN         | 47.1% | 74.3% | 65.9% |


###### Accuracy on Generated Pizza Images
|     Method       | Top 1 Accuracy | Top 5 Accuracy | Confidence Pasta > Confidence Pizza |
|:----------------:|:--------------:|:--------------:|:-----------------------------------:|
| __A-GAN 15__     | __47.3%__ | __75.9%__ | __68.0%__|
| A-GAN 15 PAGE-Net| 26.2% | 60.0% | 54.7% |
| CycleGAN         | 21.7% | 59.2% | 36.4% |


The results in tables above show that the generated images have high success rate
in the translation to the target domain. The top 1 accuracy gives us the 
percentage of all images that have been classified as the target class with 
the highest confidence by the classifier and similarly for ’Top 5’. 
The last column shows the percentages of images which has been classified as
the target class with a higher confidence, than the source class. This column 
is the most important one, as it shows the success rate of the translation to
the target domain. In both directions A-GAN 15 is significantly superior than
the other two methods with 79.9% and 68% successful translations.

An interesting observation is that A-GAN 15 generated pasta images have very 
high top 5 accuracy and success rate but a lot much lower top 1 accuracy
. This means that the pizza to pasta translation was very successful but the 
classifier confuses it with some other food category. A possible future 
improvement is to train a multi-domain A-GAN 15 in which the discriminator would
have to distinguish not only between pizza and pasta but with other categories 
as well. This would potentially improve the translation of the unique
characteristics of each food category.

## Can the generated images improve the accuracy of a classifier?

An interesting idea for unusual data augmentation is to use the training set to 
train a GAN to generate more images using image-to-image translation. This 
approach could be used as another evaluation of the generations, because if
the accuracy of the test set does not drop significantly when the generated 
images are added to the original training dataset, this means that they are
enough good not to confuse the training classifier.

The following table show the results of a Resnet 50 classifier trained on 19 food classes:

|     Dataset       | All calsses | Pizza and Pasta | Pizza | Pasta |
|:-----------------:|:-----------:|:---------------:|:-----:|:------:|
| No generations    | 83.9%       | __95.6%__       | __97.6%__| 93.6%|
| A-GAN 15          | __85.3%__   | 93.0% | 92.0%| 94.0%|
| A-GAN 15 PAGE-Net| 85.1% |       93.1% | 92.4% | 94.0%|
| CycleGAN         | 84.2% |       93.6% | 93.6% | 93.6%|
