Export a different resolutions

[ferdinandl007]

What would need to be changed so that models can be exported at different resolutions for example 640 to 320

[ferdinandl007]

As most of the models are not capable of running in real time on an iOS device with such large resolutions

[ferdinandl007]

It would be great if you could pass in a desired import resolution also change the aspect ratio from scratch to something more like an 6:4 or 19:9 aspect ratio

[Leon0402]

I haven't used other resolutions, but I believe the steps necessary would be:

Change the input images for tracing & exporting
https://github.com/dbsystel/yolov5-coreml-tools/blob/main/src/coreml_export/main.py#L308

Say 192 x 320, this would give you the three outputs <1, 3, 85, 40, 24>, <1, 3, 85, 20, 12>, <1, 3, 85, 10, 6>, which need to be processed in order to get confidence and coordinates. The last numbers are just the (width, height) / 8, (width/height) / 16, (width, height) / 32. So divided by the strides.

The feature map changes accordingly and now needs two different values for width and height:
https://github.com/dbsystel/yolov5-coreml-tools/blob/main/src/coreml_export/main.py#L44

And change the meta data to use the different width and height values:
https://github.com/dbsystel/yolov5-coreml-tools/blob/main/src/coreml_export/main.py#L96

Pass in here then both width and height:
https://github.com/dbsystel/yolov5-coreml-tools/blob/main/src/coreml_export/main.py#L127

Normalize Values here accordingly:
https://github.com/dbsystel/yolov5-coreml-tools/blob/main/src/coreml_export/main.py#L155

The output changes then as well:
https://github.com/dbsystel/yolov5-coreml-tools/blob/main/src/coreml_export/main.py#L179

Should be 25400 = 3 * (80² + 40² + 20²) and should now be 3 * (4024 + 2012 + 10*6) = 3780 I believe.

Please let me know if that works.

Also, Pull Requests are welcome! Would be great to make this more customizable by passing the width and height of the pytorch model to the conversion skript.

As most of the models are not capable of running in real time on an iOS device with such large resolutions

No problems so far :) The model is a few ms slower than the official from yolov5 in my tests, but with around 20ms per detection it's already reasonable fast I would say. I haven't tested quality differences (see ultralytics/yolov5#2526).

[ferdinandl007]

Thanks for the guidance on how to change it I will have a look at it and will send a PR when I get it working.
I noticed my inference is pretty slow when running yolov5l on 640 about 700 ms and it will crash the device when running through sustained periods of time. I get an error code com.apple.vis 3. I suspect the neural Engine runs out of memory or something as it's in the unknown error category

[ferdinandl007]

Which models were you testing which got such good performance at higher resolutions? s, m?

[Leon0402]

I run the yolov5s models and the results have been quite similar for release 2.0, 3.0, 4.0. I think I also tested it with m, which was about 40ms, but I can't quite remember. I didn't put much effort in it though to optimize it.

700ms is extremly slow. Without optimization the yolov5s model runs slower than > 100ms, but not 600ms or 700ms.

As I pointed out in the linked issue, the model doesn't run completly on the Neural Engine by default as it has problems with the pooling layer.

You might wanna modify the kernel sizes. Here is some code to get you started:
https://github.com/dbsystel/yolov5-coreml-tools/blob/main/src/coreml_export/snippets.py#L22

This is just for debugging purposes though. You should obviously fix this in your pytorch model and retrain with the changes.

This issue is, as far as I can tell, relevant to the offcial coreml model from yolov5 as well (which uses a smaller resolution). Although it didn't lead to any (noticable) speed improvements, it decreased CPU usage a lot.

[Leon0402]

Actually I recently modified the snippets code for some tests to work with the current full (pipeline) model. The current code is just for the "raw" converted model.

Not pretty, but does the job.

def fixModel(spec):
    kernelSizes = [3, 5, 7]
    poolingCount = 0 

    # print(dir(modelSpec.pipeline.models))
    #print(modelSpec.WhichOneof('Type'))

    modelSpec = spec.pipeline.models[0]


    for layer in modelSpec.neuralNetwork.layers:
        if layer.WhichOneof("layer") == "pooling":
            layer.pooling.kernelSize[:] = [kernelSizes[poolingCount], kernelSizes[poolingCount]] #[5, 5]

            layer.pooling.valid.paddingAmounts.borderAmounts[0].startEdgeSize = kernelSizes[poolingCount] // 2#2
            layer.pooling.valid.paddingAmounts.borderAmounts[0].endEdgeSize =  kernelSizes[poolingCount] // 2#2
            layer.pooling.valid.paddingAmounts.borderAmounts[1].startEdgeSize = kernelSizes[poolingCount] // 2# 2
            layer.pooling.valid.paddingAmounts.borderAmounts[1].endEdgeSize = kernelSizes[poolingCount] // 2#2

            poolingCount += 1

    print(poolingCount)

    #spec.pipeline.models[0] = modelSpec

    ct.models.utils.save_spec(spec, "output/models/original-fixed.mlmodel")

[ferdinandl007]

ha ok I will give it a try modifying the kernel sizes,
Also if you ever want to train YOLO with sync_batchnorm which is good when training on multiple machines or GPUs the layers will need to be removed when exporting might be worth adding that to the script as well :)
def revert_sync_batchnorm(module): # this is very similar to the function that it is trying to revert: # https://github.com/pytorch/pytorch/blob/c8b3686a3e4ba63dc59e5dcfe5db3430df256833/torch/nn/modules/batchnorm.py#L679 module_output = module if isinstance(module, torch.nn.modules.batchnorm.SyncBatchNorm): module_output = nn.BatchNorm2d( module.num_features, module.eps, module.momentum, module.affine, module.track_running_stats, ) if module.affine: with torch.no_grad(): module_output.weight = module.weight module_output.bias = module.bias module_output.running_mean = module.running_mean module_output.running_var = module.running_var module_output.num_batches_tracked = module.num_batches_tracked if hasattr(module, "qconfig"): module_output.qconfig = module.qconfig for name, child in module.named_children(): module_output.add_module(name, revert_sync_batchnorm(child)) del module return module_output

[ferdinandl007]

@Leon0402 I was wondering how can I add different scales to X and Y they don't see any easy way of achieving this with this method builder.add_scale

Sorry for the dumb question I'm new to coreMl and the documentation is not great :/

[Leon0402]

You can specify a shape_scale according to the documentation. So specify the option that suits you and then pass a tuple or list as the scale.

Have a look at https://github.com/apple/coremltools/blob/main/mlmodel/format/NeuralNetwork.proto#L27 for a mapping.

I'm not quite sure as I haven't worked on my source code lately, but presumably [C,H,W] should be the desired shape.

You're right the documentation is improveable and often these functions are not very flexible. In fact there seems to be a new method to actually write coreml neural layers, which is much easier at the first glance. On the other hand the beta of CoreML 4.0 was literally just released when I started working on this. So I didn't want to use too many experimental features, there were already enough bugs in the first version I had to deal with :-)

[usmanqureshi94]

Any luck finding the solution for non squared input size. I tried the above steps but it doesn't work.
I get complication error while running inference on macOS. I am not sure if I updated add_scale correctly

builder.add_scale(name=f"normalize_coordinates_{outputName}", input_name=f"{outputName}_raw_coordinates", output_name=f"{outputName}_raw_normalized_coordinates", W=torch.tensor([3,1/image_height, 1/image_width]).numpy(), b=0, has_bias=False, shape_scale = [3])

[Leon0402]

I don't think shape_scale = [3] is valid, at least not according to https://github.com/apple/coremltools/blob/6ec26d282323a4aec33252731e31fdb26569bfb2/coremltools/models/neural_network/builder.py#L2070

What's the error message you get?

It would be helpful if you fork the repo and do the changes, then provide here a model where I can test your repo with. This makes it easier to see where you might have done a mistake.