Skip to content

Commit 3f22f1c

Browse files
gauravpradeepgauravpradeep
committed
merging multiscan
2 parents 7545e42 + 2d1b724 commit 3f22f1c

31 files changed

+1593
-23
lines changed

‎DATA.md

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ We list the available data used in the current version of CrossOver in the table
1111
| ScanNet | `[point, rgb, cad, referral]` | `[point, rgb, floorplan, referral]` | ❌ | ✅ |
1212
| 3RScan | `[point, rgb, referral]` | `[point, rgb, referral]` | ✅ | ✅ |
1313
| ARKitScenes | `[point, rgb, referral]` | `[point, rgb, referral]` | ❌ | ✅ |
14+
| MultiScan | `[point, rgb, referral]` | `[point, rgb, referral]` | ❌ | ✅ |
1415

1516

1617
We detail data download and release instructions for preprocessing with scripts for ScanNet + 3RScan.
@@ -112,6 +113,39 @@ Scan3R/
112113
| │ └── sel_cams_on_mesh.png (visualisation of the cameras selected for computing RGB features per scan)
113114
| └── ...
114115
```
116+
### MultiScan
117+
118+
#### Running preprocessing scripts
119+
Adjust the path parameters of `MultiScan` in the config files under `configs/preprocess`. Run the following (after changing the `--config-path` in the bash file):
120+
121+
```bash
122+
$ bash scripts/preprocess/process_multiscan.sh
123+
```
124+
125+
Our script for MultiScan dataset performs the following additional processing:
126+
127+
- 3D-to-2D projection for 2D segmentation and stores as `gt-projection-seg.pt` for each scan.
128+
129+
Post running preprocessing, the data structure should look like the following:
130+
131+
```
132+
MultiScan/
133+
├── objects_chunked/ (object data chunked into hdf5 format for instance baseline training)
134+
| ├── train_objects.h5
135+
| └── val_objects.h5
136+
├── scans/
137+
| ├── scene_00000_00/
138+
| │ ├── gt-projection-seg.pt -> 3D-to-2D projected data consisting of framewise 2D instance segmentation
139+
| │ ├── data1D.pt -> all 1D data + encoded (object referrals + BLIP features)
140+
| │ ├── data2D.pt -> all 2D data + encoded (RGB + floorplan + DinoV2 features)
141+
| │ ├── data2D_all_images.pt (RGB features of every image of every scan)
142+
| │ ├── data3D.pt -> all 3D data + encoded (Point Cloud + I2PMAE features - object only)
143+
| │ ├── object_id_to_label_id_map.pt -> Instance ID to NYU40 Label mapped
144+
| │ ├── objectsDataMultimodal.pt -> object data combined from data1D.pt + data2D.pt + data3D.pt (for easier loading)
145+
| │ └── sel_cams_on_mesh.png (visualisation of the cameras selected for computing RGB features per scan)
146+
| └── ...
147+
```
148+
115149

116150
### ARKitScenes
117151

@@ -144,4 +178,4 @@ ARKitScenes/
144178
| │ ├── objectsDataMultimodal.pt -> object data combined from data1D.pt + data2D.pt + data3D.pt (for easier loading)
145179
| │ └── sel_cams_on_mesh.png (visualisation of the cameras selected for computing RGB features per scan)
146180
| └── ...
147-
```
181+
```

‎README.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,8 @@ See [DATA.MD](DATA.md) for detailed instructions on data download, preparation a
118118
| Scannet | `[point, rgb, cad, referral]` | `[point, rgb, floorplan, referral]` | ❌ | ✅ |
119119
| 3RScan | `[point, rgb, referral]` | `[point, rgb, referral]` | ✅ | ✅ |
120120
| ARKitScenes | `[point, rgb, referral]` | `[point, rgb, referral]` | ❌ | ✅ |
121+
| MultiScan | `[point, rgb, referral]` | `[point, rgb, referral]` | ❌ | ✅ |
122+
121123

122124
> To run our demo, you only need to download generated embedding data; no need for any data preprocessing.
123125
@@ -134,7 +136,7 @@ Various configurable parameters:
134136
- `--database_path`: Path to the precomputed embeddings of the database scenes downloaded before (eg: `./release_data/embed_scannet.pt`).
135137
- `--query_modality`: Modality of the query scene, Options: `point`, `rgb`, `floorplan`, `referral`
136138
- `--database_modality`: Modality used for retrieval. Same options as above.
137-
- `--ckpt`: Path to the pre-trained scene crossover model checkpoint (details [here](#checkpoints)), example_path: `./checkpoints/scene_crossover_scannet+scan3r.pth/`).
139+
- `--ckpt`: Path to the pre-trained scene crossover model checkpoint (details [here](#checkpoints)), example_path: `./checkpoints/scene_crossover_scannet+scan3r.pth/`.
138140

139141
For embedding and pre-trained model download, refer to [generated embedding data](DATA.md#generated-embedding-data) and [checkpoints](#checkpoints) sections.
140142

‎TRAIN.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ $ bash scripts/train/train_instance_crossover.sh
2121
```
2222

2323
#### Train Scene Retrieval Pipeline
24-
Adjust path/configuration parameters in `configs/train/train_scene_crossover.yaml`. You can also add your customised dataset or choose to train on Scannet, 3RScan & ARKitScenes or any combination of the same. Run the following:
24+
Adjust path/configuration parameters in `configs/train/train_scene_crossover.yaml`. You can also add your customised dataset or choose to train on Scannet, 3RScan, MultiScan & ARKitScenes or any combination of the same. Run the following:
2525

2626
```bash
2727
$ bash scripts/train/train_scene_crossover.sh

‎configs/evaluation/eval_instance.yaml

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,13 +53,23 @@ data :
5353
max_object_len : 150
5454
voxel_size : 0.02
5555

56+
MultiScan:
57+
base_dir : /media/sayan/Expansion/data/datasets/MultiScan
58+
process_dir : ${data.process_dir}/MultiScan
59+
processor3D : MultiScan3DProcessor
60+
processor2D : MultiScan2DProcessor
61+
processor1D : MultiScan1DProcessor
62+
avail_modalities : ['point', 'cad', 'rgb', 'referral']
63+
max_object_len : 150
64+
voxel_size : 0.02
65+
5666
task:
5767
name : InferenceObjectRetrieval
5868
InferenceObjectRetrieval:
5969
val : [Scannet]
6070
modalities : ['rgb', 'point', 'cad', 'referral']
6171
scene_modalities : ['rgb', 'point', 'referral', 'floorplan']
62-
ckpt_path : /drive/dumps/multimodal-spaces/runs/release_runs/instance_crossover_scannet+scan3r+arkit.pth
72+
ckpt_path : /drive/dumps/multimodal-spaces/runs/release_runs/instance_crossover_scannet+scan3r+multiscan+arkit.pth
6373

6474

6575
inference_module: ObjectRetrieval

‎configs/evaluation/eval_scene.yaml

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,12 @@ data :
4949
processor3D : ARKitScenes3DProcessor
5050
processor2D : ARKitScenes2DProcessor
5151
processor1D : ARKitScenes1DProcessor
52+
MultiScan:
53+
base_dir : /media/sayan/Expansion/data/datasets/MultiScan
54+
process_dir : ${data.process_dir}/MultiScan
55+
processor3D : MultiScan3DProcessor
56+
processor2D : MultiScan2DProcessor
57+
processor1D : MultiScan1DProcessor
5258
avail_modalities : ['point', 'cad', 'rgb', 'referral']
5359
max_object_len : 150
5460
voxel_size : 0.02
@@ -59,7 +65,7 @@ task:
5965
val : [Scannet]
6066
modalities : ['rgb', 'point', 'cad', 'referral']
6167
scene_modalities : ['rgb', 'point', 'referral', 'floorplan'] #, 'point']
62-
ckpt_path : /drive/dumps/multimodal-spaces/runs/release_runs/scene_crossover_scannet+scan3r+arkit.pth
68+
ckpt_path : /drive/dumps/multimodal-spaces/runs/release_runs/scene_crossover_scannet+scan3r+multiscan+arkit.pth
6369

6470
inference_module: SceneRetrieval
6571
model:

‎configs/preprocess/process_1d.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,17 @@ data:
3232
processor2D : ARKitScenes2DProcessor
3333
processor1D : ARKitScenes1DProcessor
3434
skip_frames : 1
35+
36+
MultiScan:
37+
base_dir : /media/sayan/Expansion/data/datasets/MultiScan
38+
process_dir : ${data.process_dir}/MultiScan
39+
processor3D : MultiScan3DProcessor
40+
processor2D : MultiScan2DProcessor
41+
processor1D : MultiScan1DProcessor
42+
skip_frames : 1
43+
44+
Shapenet:
45+
base_dir : /drive/datasets/Shapenet/ShapeNetCore.v2/
3546

3647
modality_info:
3748
1D :

‎configs/preprocess/process_2d.yaml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,13 @@ data:
3333
processor3D : ARKitScenes3DProcessor
3434
processor2D : ARKitScenes2DProcessor
3535
processor1D : ARKitScenes1DProcessor
36+
37+
MultiScan:
38+
base_dir : /media/sayan/Expansion/data/datasets/MultiScan
39+
process_dir : ${data.process_dir}/MultiScan
40+
processor3D : MultiScan3DProcessor
41+
processor2D : MultiScan2DProcessor
42+
processor1D : MultiScan1DProcessor
3643
skip_frames : 1
3744

3845
modality_info:

‎configs/preprocess/process_3d.yaml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,13 @@ data:
3030
processor3D : ARKitScenes3DProcessor
3131
processor2D : ARKitScenes2DProcessor
3232
processor1D : ARKitScenes1DProcessor
33+
34+
MultiScan:
35+
base_dir : /media/sayan/Expansion/data/datasets/MultiScan
36+
process_dir : ${data.process_dir}/MultiScan
37+
processor3D : MultiScan3DProcessor
38+
processor2D : MultiScan2DProcessor
39+
processor1D : MultiScan1DProcessor
3340
skip_frames : 1
3441

3542
modality_info:

‎configs/preprocess/process_multimodal.yaml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,14 @@ data:
3535
processor3D : ARKitScenes3DProcessor
3636
processor2D : ARKitScenes2DProcessor
3737
processor1D : ARKitScenes1DProcessor
38+
39+
MultiScan:
40+
base_dir : /media/sayan/Expansion/data/datasets/MultiScan
41+
process_dir : ${data.process_dir}/MultiScan/
42+
chunked_dir : ${data.process_dir}/MultiScan/objects_chunked
43+
processor3D : Scan3R3DProcessor
44+
processor2D : Scan3R2DProcessor
45+
processor1D : Scan3R1DProcessor
3846
avail_modalities : ['point', 'rgb', 'referral']
3947

4048
modality_info:

‎configs/train/train_instance_baseline.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,17 @@ data :
5555
max_object_len : 150
5656
voxel_size : 0.02
5757

58+
MultiScan:
59+
base_dir : /media/sayan/Expansion/data/datasets/Multiscan
60+
process_dir : ${data.process_dir}/MultiScan/
61+
chunked_dir : ${data.process_dir}/MultiScan/objects_chunked
62+
processor3D : MultiScan3DProcessor
63+
processor2D : MultiScan2DProcessor
64+
processor1D : MultiScan1DProcessor
65+
avail_modalities : ['point', 'rgb', 'referral']
66+
max_object_len : 150
67+
voxel_size : 0.02
68+
5869
task:
5970
name : ObjectLevelGrounding
6071
ObjectLevelGrounding :

0 commit comments

Comments
 (0)