Using a pre-trained ResNet neural network for image classification¶
Neural networks are a convenient flexible algorithm that can be trained computationally. With a successful training process, the same neural network can be used in many different tasks. This, for example, is the famous family of neural networks ResNet, created for image classification. In part or in whole, such neural networks are used in a wide variety of tasks where it is necessary to work with numerical representations of images, from graphical databases to style transfer.
In this example, we will run a neural network ResNet of shallow depth (18 layers) for image classification. We will show the whole chain of data preparation and output processing, which for any image will give us a more or less appropriate text label characterising the object depicted in the image.
Preparatory work¶
The Tape object of the Umlaut library , at the moment, is the main computational container where a neural network from the ONNX format can be unpacked. This startup mechanism is likely to be changed in the near future, as the ONNX library is in the process of being updated.
Pkg.add(["Umlaut", "ONNX"])
import Pkg; Pkg.add("Umlaut", io=devnull)
import Umlaut: Tape, play!
Of course, we will also need the libraries built into Engee to work with the ONNX format, where pre-trained neural networks are often stored, and a library to work with images.
using ONNX
using Images
Let's install the working folder
cd( @__DIR__ )
Object classes in ImageNet¶
The names of all object classes that our pre-trained neural network can recognise are represented as an ordered vector and are loaded from the following file.
include( "imagenet_classes.jl" );
Input-output functions¶
Let's write three simple auxiliary functions to feed data into the neural network and process the results:
- Loading an image: we scale it to a size of 244*244 (normalisation and edge cropping with aspect ratio preservation would be a welcome addition)
- Sorting predictions: the neural network returns a vector of numbers that denote the probability that a particular class from the ImageNet dataset is observed in the image. Let's select
kof the most likely represented classes - The Shell for these functions allows you to load an image and output
kof the most likely predictions in a single action
# Загрузка изображения из файла
function imread(path::AbstractString; sz=(224,224))
img = Images.load(path);
img = imresize(img, sz);
x = convert(Array{Float32}, channelview(img))
# Заменим порядок слоев: CHW -> WHC
x = permutedims(x, (3, 2, 1))
return x
end
# Выдача индексов первых k предсказаний
function maxk(a, k)
b = partialsortperm(a, 1:k, rev=true)
return collect(zip(b, a[b]))
end
# Загрузка изображения и выдача десяти наиболее вероятных классов в убывающем порядке
function test_image(tape::Tape, path::AbstractString)
x = imread(path)
x = reshape(x, size(x)..., 1)
y = play!(tape, x)
y = reshape(y, size(y, 1))
top = maxk(y, 10)
classes = []
for (i, (idx, val)) in enumerate(top)
name = IMAGENET_CLASSES[idx - 1]
classes = [classes; "$i: $name ($val)"]
end
return join(classes, "\n")
end
Download the neural network ResNet18¶
We will be using a neural network that lies in a file with the extension *.onnx. There are libraries that allow you to create and load this neural network using even higher-level commands (e.g., the Metalhead.jl library from the FluxML collection), but for now we'll do it without additional libraries.
The pre-trained neural network is already in the specified directory, so the command will execute without re-downloading.
path = "resnet18.onnx"
if !isfile(path)
download("https://github.com/onnx/models/raw/main/vision/classification/resnet/model/resnet18-v1-7.onnx", path)
end
# Создадим пустую матрицу на месте которой будет входное изображение
img = rand( Float32, 224, 224, 3, 1 )
# Загружаем модель в виде объекта Umlaut.Tape
resnet = ONNX.load( path, img );
Let's download some images¶
If they have already been downloaded, similarly, they will not be downloaded again
path = "data/"
goose_path = download( "https://upload.wikimedia.org/wikipedia/commons/3/3f/Snow_goose_2.jpg", path*"goose.jpg");
dog_path = download( "https://farm4.staticflickr.com/1301/4694470234_6f27a4f602_o.jpg", path*"dog.jpg");
plane_path = download( "https://upload.wikimedia.org/wikipedia/commons/thumb/c/c9/Rossiya%2C_RA-89043%2C_Sukhoi_Superjet_100-95B_%2851271265892%29.jpg/1024px-Rossiya%2C_RA-89043%2C_Sukhoi_Superjet_100-95B_%2851271265892%29.jpg", path*"plane.jpg");
Categorising images¶
display( load(plane_path)[1:5:end, 1:5:end] )
print( test_image( resnet, plane_path ))
display( load(goose_path)[1:5:end, 1:5:end] )
print( test_image( resnet, goose_path ))
display( load(dog_path)[1:5:end, 1:5:end] )
print( test_image( resnet, dog_path ))
Conclusion¶
We have shown that it is not difficult to download a neural network in Engee and perform computations with it.
This mechanism allows us to organise a complex information processing pipeline consisting of high-level components. In particular, we can entrust some stages of information processing to pre-trained neural networks.
Bibliography: https://github.com/FluxML/ONNX.jl