resolving conflicts

Author: minerva-ml

README.md

@@ -10,59 +10,89 @@ We are building entirely open solution to this competition. Specifically:
 1. **Learning from the process** - updates about new ideas, code and experiments is the best way to learn data science. Our activity is especially useful for people who wants to enter the competition, but lack appropriate experience.
 1. Encourage more Kagglers to start working on this competition.
 1. Deliver open source solution with no strings attached. Code is available on our [GitHub repository :computer:](https://github.com/neptune-ml/open-solution-googleai-object-detection). This solution should establish solid benchmark, as well as provide good base for your custom ideas and experiments. We care about clean code :smiley:
-1. We are opening our experiments as well: everybody can have **live preview** on our experiments, parameters, code, etc. Check: [Google-AI-Object-Detection-Challenge :chart_with_upwards_trend:](https://app.neptune.ml/neptune-ml/Google-AI-Object-Detection-Challenge).
+1. We are opening our experiments as well: everybody can have **live preview** on our experiments, parameters, code, etc. Check: [Google-AI-Object-Detection-Challenge :chart_with_upwards_trend:](https://app.neptune.ml/neptune-ml/Google-AI-Object-Detection-Challenge) and images below:
+
+| UNet training monitor :bar_chart: | Predicted bounding boxes :bar_chart: |
+|:---|:---|
+|[![unet-training-monitor](https://gist.githubusercontent.com/kamil-kaczmarek/b3b939797fb39752c45fdadfedba3ed9/raw/19272701575bca235473adaabb7b7c54b2416a54/gai-1.png)](https://app.neptune.ml/-/dashboard/experiment/f945da64-6dd3-459b-94c5-58bc6a83f590)|[![predicted-bounding-boxes](https://gist.githubusercontent.com/kamil-kaczmarek/b3b939797fb39752c45fdadfedba3ed9/raw/19272701575bca235473adaabb7b7c54b2416a54/gai-2.png)](https://app.neptune.ml/-/dashboard/experiment/c779468e-d3f7-44b8-a3a4-43a012315708)|
+
+## Disclaimer
+In this open source solution you will find references to the [neptune.ml](https://neptune.ml). It is free platform for community Users, which we use daily to keep track of our experiments. Please note that using neptune.ml is not necessary to proceed with this solution. You may run it as plain Python script :snake:.
+
+# How to start?
+## Learn about our solutions
+1. Check [Kaggle forum](https://www.kaggle.com/c/google-ai-open-images-object-detection-track/discussion/62895) and participate in the discussions.
+1. Check our [Wiki pages :dolphin:](https://github.com/neptune-ml/open-solution-googleai-object-detection/wiki), where we describe our work. Below are link to specific solutions:
+
+| link to code| link to description |
+|:---:|:---:|
+|[solution-1](https://github.com/neptune-ml/open-solution-googleai-object-detection/tree/solution-1)|[palm-tree :palm_tree:](https://github.com/neptune-ml/open-solution-googleai-object-detection/wiki/RetinaNet-with-sampler)|
 
 ## Dataset for this competition
 This competition is special, because it used [Open Images Dataset V4](https://storage.googleapis.com/openimages/web/index.html), which is quite large: `>1.8M` images and `>0.5TB` :astonished: To make it more approachable, we are hosting entire dataset in the neptune's public directory :sunglasses:. **You can use this dataset in [neptune.ml](https://neptune.ml) with no additional setup :+1:.**
 
-## Learn more about our solutions
-[Kaggle discussion](https://www.kaggle.com/c/google-ai-open-images-object-detection-track/discussion) is our primary way of communication, however, we are also documenting our work on the [Wiki pages :blue_book:](https://github.com/neptune-ml/open-solution-googleai-object-detection/wiki). Click on the dolphin to get started [:dolphin:](https://github.com/neptune-ml/open-solution-googleai-object-detection/wiki).
-
-## Disclaimer
-In this open source solution you will find references to the [neptune.ml](https://neptune.ml). It is free platform for community Users, which we use daily to keep track of our experiments. Please note that using neptune.ml is not necessary to proceed with this solution. You may run it as plain Python script :wink:.
+## Start experimenting with ready-to-use code
+You can jump start your participation in the competition by using our starter pack. Installation instruction below will guide you through the setup.
 
 ## Installation
 ### Fast Track
-1. Clone repository and install requirements (check _requirements.txt_)
-1. Register to the [neptune.ml](https://neptune.ml/login) _(if you wish to use it)_
-1. Run experiment:
+1. Clone repository, install requirements (check _requirements.txt)
+
+```bash
+pip3 install -r requirements.txt
+```
+
+2. Register to the [neptune.ml](https://neptune.ml/login) _(if you wish to use it)_ and create your project, for example Google-AI-Object-Detection-Challenge.
+3. Train RetinaNet:
+
+:hamster:
+```bash
+neptune send --worker m-4p100 \
+--environment pytorch-0.3.1-gpu-py3 \
+--config configs/neptune.yaml \
+main.py train --pipeline_name retinanet
+```
 
 :trident:
 ```bash
-neptune run will appear here soon :)
+neptune run main.py train --pipeline_name retinanet
 ```
 
 :snake:
 ```bash
-python command will appear here soon :)
+python main.py -- train --pipeline_name retinanet
 ```
 
-### Step by step
-1. Clone this repository
-```bash
-git clone https://github.com/neptune-ml/open-solution-googleai-object-detection.git
+4. Evaluate/Predict RetinaNet:
+
+**Note** in case of memory trouble go to `neptune.yaml` and change `batch_size_inference: 1`
+
+:hamster:
+With cloud environment you need to change the experiment directory to the one that you have just trained. Let's assume that your experiment id was `GAI-14`. You should go to `neptune.yaml` and change:
+
+```yaml
+ experiment_dir: /output/experiment
+ clone_experiment_dir_from: /input/GAI-14/output/experiment
 ```
-2. Install requirements in your Python3 environment
+
 ```bash
-pip3 install requirements.txt
+neptune send --worker m-4p100 \
+--environment pytorch-0.3.1-gpu-py3 \
+--config configs/neptune.yaml \
+--input /GAI-14 \
+main.py evaluate_predict --pipeline_name retinanet --chunk_size 100
 ```
-3. Register to the [neptune.ml](https://neptune.ml/login) _(if you wish to use it)_
-4. Update data directories in the [neptune.yaml](https://github.com/neptune-ml/open-solution-googleai-object-detection/blob/master/neptune.yaml) configuration file
-5. Run experiment:
 
 :trident:
 ```bash
-neptune login
-neptune run will appear here soon :)
+neptune run main.py train --pipeline_name retinanet --chunk_size 100
 ```
 
 :snake:
 ```bash
-python command will appear here soon :)
+python main.py -- train --pipeline_name retinanet --chunk_size 100
 ```
 
-6. collect submit from `experiment_directory` specified in the [neptune.yaml](https://github.com/neptune-ml/open-solution-googleai-object-detection/blob/master/neptune.yaml)
-
 ## Get involved
 You are welcome to contribute your code and ideas to this open solution. To get started:
 1. Check [competition project](https://github.com/neptune-ml/open-solution-googleai-object-detection/projects/1) on GitHub to see what we are working on right now.
@@ -72,6 +102,6 @@ You are welcome to contribute your code and ideas to this open solution. To get
 
 ## User support
 There are several ways to seek help:
-1. [Kaggle discussion](https://www.kaggle.com/c/google-ai-open-images-object-detection-track/discussion) is our primary way of communication.
+1. [Kaggle discussion](https://www.kaggle.com/c/google-ai-open-images-object-detection-track/discussion/62895) is our primary way of communication.
 1. Read project's [Wiki](https://github.com/neptune-ml/open-solution-googleai-object-detection/wiki), where we publish descriptions about the code, pipelines and supporting tools such as [neptune.ml](https://neptune.ml).
 1. Submit an [issue]((https://github.com/neptune-ml/open-solution-googleai-object-detection/issues)) directly in this repo.

configs/neptune.yaml

@@ -1,14 +1,14 @@
-project: YOUR_PROJECT_NAME
+project: USERNAME/googleai-object-detection
 
-name: google AI object detection
+name: Google AI object detection
 tags: [solution-1]
 
 metric:
  channel: 'MAP'
  goal: maximize
 
 #Comment out if not in Cloud Environment
-#pip-requirements-file: requirements.txt
+pip-requirements-file: requirements.txt # Comment out if Local execution
 
 exclude:
  - .git
@@ -23,18 +23,18 @@ exclude:
 
 parameters:
 # Data Paths
- train_imgs_dir: ''
- test_imgs_dir: ''
- annotations_filepath: ''
- annotations_human_labels_filepath: ''
- bbox_hierarchy_filepath: ''
- valid_ids_filepath: ''
- sample_submission: ''
- experiment_dir: ''
- class_mappings_filepath: ''
- metadata_filepath: ''
+ train_imgs_dir: /public/datasets/open-images-dataset-v4/bounding-boxes/train
+ test_imgs_dir: /public/datasets/open-images-dataset-v4/bounding-boxes/test_challenge_2018
+ annotations_filepath: /public/challenges/google-ai-open-images-object-detection-track/annotations/challenge-2018-train-annotations-bbox.csv
+ annotations_human_labels_filepath: /public/challenges/google-ai-open-images-object-detection-track/annotations/challenge-2018-train-annotations-human-imagelabels.csv
+ bbox_hierarchy_filepath: /public/challenges/google-ai-open-images-object-detection-track/metadata/bbox_labels_500_hierarchy.json
+ class_mappings_filepath: /public/challenges/google-ai-open-images-object-detection-track/metadata/challenge-2018-class-descriptions-500.csv
+ valid_ids_filepath: /public/challenges/google-ai-open-images-object-detection-track/metadata/challenge-2018-image-ids-valset-od.csv
+ sample_submission: /public/challenges/google-ai-open-images-object-detection-track/sample_submission.csv
+ experiment_dir: /output/experiment
+ clone_experiment_dir_from: '' #When running eval specify this as for example /input/GAI-14/output/experiment
 
-# Execution
+ # Execution
  clean_experiment_directory_before_training: 1
  num_workers: 4
  num_threads: 4
@@ -62,24 +62,24 @@ parameters:
 
 # Retina parameters (multi-output)
  encoder_depth: 50
- num_classes: 500
+ num_classes: 10
  pretrained_encoder: 1
  pi: 0.01
  aspect_ratios: '[1/2., 1/1., 2/1.]'
  scale_ratios: '[1., pow(2,1/3.), pow(2,2/3.)]'
 
 # Training schedule
  epochs_nr: 100
- batch_size_train: 1
- batch_size_inference: 1
+ batch_size_train: 8
+ batch_size_inference: 8
  lr: 0.00001
  momentum: 0.9
  gamma: 1.0
  patience: 30
  lr_factor: 0.3
  lr_patience: 30
- training_sample_size: 50000
- validation_sample_size: 10000
+ training_sample_size: 10000
+ validation_sample_size: 1000
 
 # Regularization
  use_batch_norm: 1

notebooks/dataset_exploration.ipynb

@@ -11,8 +11,8 @@
  "import glob\n",
  "import pandas as pd\n",
  "\n",
- "DIRPATH = '/mnt/ml-team/open-images-v4/bounding-boxes'\n",
- "DIRPATH_COMPETITION = '/mnt/ml-team/minerva/open-solutions/googleai-object-detection/data'"
+ "DIRPATH = '/PATH/TO/open-images-v4/bounding-boxes'\n",
+ "DIRPATH_COMPETITION = '/PATH/TO/DATA'"
  ]
  },
  {