Primary processing of engineering data
Introduction
Modern engineering is inconceivable without data processing: sensor readings, acoustic diagnostic signals, measurement results from test benches. All this information comes in diverse formats, from simple text files and Excel spreadsheets to specialized binary formats like MAT and WAV. The ability to correctly read, inspect, transform, and visualize such data is the basic skill of a research engineer and precedes any meaningful analysis.
This paper is a practical introduction to engineering data and signal processing in Engee. End-to-end examples demonstrate the full cycle of working with information: downloading from common formats (CSV, XLSX, TXT, MAT, WAV, PNG), initial inspection and type control, timestamp conversion, time series visualization, spectral analysis using fast Fourier transform, signal compression to a compact set of features, as well as repackaging the set data in the required structure.
Energy consumption data from a steel mill and acoustic signals from bearing assemblies for various types of defects are used as meaningful examples. Mastering these techniques forms a holistic view of the preparation of engineering data for modeling and diagnostics.
Joining libraries
We'll connect libraries for working with data, signals, and visualization. StatsBase and Statistics will provide statistical functions. CSV, XLSX, and MAT will allow you to read data from the appropriate formats. WAV is for audio files. Images - for uploading images. DSP and FFTW are for spectral analysis.
for pkg in (["Statistics", "StatsBase", "StatsPlots", "CSV", "DataFrames", "Dates", "XLSX", "MAT", "WAV", "Images", "FileIO", "DSP", "FFTW", "DelimitedFiles"])
try
eval(Meta.parse("using $pkg"))
catch
Pkg.add(pkg)
eval(Meta.parse("using $pkg"))
end
end
include("media_player.jl");
Reading a CSV file with energy consumption data
Let's start with the most common engineering data format, CSV. Let's read the file with the energy consumption indicators of the steel company. The CSV.read function returns a DataFrame, a tabular structure where columns are available by name. We will immediately deduce the dimension in order to understand the scale of the data.
df_energy = CSV.read("Steel_Industry.csv", DataFrame)
println("Data dimension: ", size(df_energy))
println("Column Names: ", names(df_energy))
Initial data inspection
The first rule of an engineer when receiving data is to look at it through his eyes. We will display the first and last rows, column types. This will help to detect problems: omissions, incorrect types, unexpected values.
println("The first 10 lines:")
first(df_energy, 10)
println("\nТипы данных столбцов:")
[names(df_energy) eltype.(eachcol(df_energy))]
Date and time conversion
In engineering data, timestamps are often stored as strings. Converting the date column to the DateTime format. After the conversion, we can calculate the date range and the average measurement step. Add a new DateTime column to the DataFrame.
df_energy.DateTime = DateTime.(df_energy.date, dateformat"dd/mm/yyyy HH:MM")
println("Range: ", minimum(df_energy.DateTime), " — ", maximum(df_energy.DateTime))
step_sec = mean([Dates.value(dt)/1000 for dt in diff(df_energy.DateTime)])
println("The middle step: ", round(step_sec, digits=2), " sec")
Visualization of a time series
A graph is the best way to understand the behavior of a system. Let's build a graph of energy consumption for the entire period. On the X-axis is time, on the Y - kilowatt-hours. We see cyclical patterns and outliers.
plot(df_energy.DateTime, df_energy.Usage_kWh,
xlabel="Time", ylabel="kWh",
title="Energy consumption",
linewidth=0.5, legend=false, size=(900, 350))
Reading data from a MAT file
The MATLAB (.mat) format is a standard in engineering practice. It can store signals, matrices, and structures. Let's read the file with the acoustic signals of the bearing. The matread function returns a dictionary, where the keys are variable names and the values are matrices.
подшипник = matread("Bearing.mat")
println("The keys are in the file: ", keys(подшипник))
println("The signal size is normal: ", size(подшипник["normal"]))
Converting MAT data to a vector
The data from the MAT file is stored as 1×N matrices. We transform them into one-dimensional vectors using the vec function. Now it's a regular Julia array of numbers that you can perform mathematical operations with.
норма = vec(подшипник["normal"])
ролик = vec(подшипник["roller"])
println("Data type: ", typeof(норма))
println("Signal length: ", length(норма))
println("\nПервые 5 отсчётов нормального сигнала: \n", норма[1:5])
Reading a WAV audio file
Acoustic data is often stored in audio formats. The wavread function returns an array of samples and a sampling rate. Unlike MAT, here the data is immediately represented as a vector.
сигнал_wav, fs_wav = WAV.wavread("Подшипник_1.wav")
println("Sampling rate: ", fs_wav, " Hz")
println("Duration: ", round(length(сигнал_wav)/fs_wav, digits=2), " sec")
Audio file playback in Engee
We will reproduce the acoustic signal of the bearing using the built-in audio player. To play an audio file, run this code cell, and playback will start automatically.
media_player("$(@__DIR__)/Подшипник_1.wav", mode="audio")
Spectral analysis via FFT
The fast Fourier transform translates the signal from the time domain to the frequency domain. It is a key tool for vibration and acoustic analysis. We center the signal, apply FFT, and take the module for the amplitude spectrum. We will plot only the first half (frequencies from 0 to Nyquist).
сигнал = норма[1:4000] .- mean(норма[1:4000])
спектр = abs.(fft(сигнал))
N = length(сигнал)
частоты = (0:N÷2-1) * 10000 / N
plot(частоты, спектр[1:N÷2],
xlabel="Frequency, Hz", ylabel="The amplitude",
title="Acoustic spectrum of the bearing",
linewidth=1, legend=false, size=(800, 350))
Data compression via FFT
One of the practical tasks is data compression. Let's look at the spectrum and determine in which range the main energy of the signal is concentrated. Then we will leave only a few key frequency components - this will be a compressed representation. Instead of 4000 signal samples, you can store 20 numbers.
амплитуды = спектр[1:N÷2]
энергия_общая = sum(амплитуды.^2)
кум_энергия = cumsum(амплитуды.^2) / энергия_общая
plot(частоты, кум_энергия,
xlabel="Frequency, Hz", ylabel="Stored energy",
title="Cumulative energy of the spectrum",
linewidth=2, legend=false, size=(800, 350))
hline!([0.95], linestyle=:dash, label="95% of the energy")
Selecting informative frequency bands
Let's divide the frequency range into 20 equal bands and calculate the average amplitude in each. This will give a compact representation of the signal - 20 numbers that can be stored in a table.
кол_полос = 20
полосы = range(0, 5000, length=кол_полос+1)
признаки_fft = zeros(кол_полос)
for i in 1:кол_полос
маска = (частоты .>= полосы[i]) .& (частоты .< полосы[i+1])
if sum(маска) > 0
признаки_fft[i] = mean(амплитуды[маска])
end
end
bar(1:кол_полос, признаки_fft,
xlabel="Band number", ylabel="Average amplitude",
title="Compressed signal representation (20 bands)",
legend=false, size=(700, 300))
Reading an image
Let's read the image file. Let's display the image type and its size.
изобр = Images.load("image.png")
println("Image Type: ", typeof(изобр))
println("Size: ", size(изобр))
Let's display the image.
изобр
Converting an image to a numeric matrix
Convert the image into a matrix of numbers.
каналы = Float64.(Images.channelview(изобр))
каналы = permutedims(каналы, (2, 3, 1))
The image is represented by a 954×923×4 three-dimensional array. The first two dimensions are 954 rows and 923 columns of pixels. The third dimension consists of four color channels: red, green, blue, and an alpha transparency channel. Each value ranges from 0 to 1, where for color channels 0 means no color, 1 means maximum intensity, and for the alpha channel 1 means full opacity.
Reading a text file with data
Text files are the simplest storage format. Let's read the txt file with numerical data on ambient temperature, determine the number of values, the range, and output the first 5 values.
температура = vec(readdlm("температура.txt"))
println("Number of values: ", length(температура))
println("Range: ", minimum(температура), " ... ", maximum(температура), " °C")
println("The first 5 values: ", температура[1:5])
Reading data from Excel
Excel files are widely distributed in the industry. Similarly, let's read the xlsx file with data on ambient humidity.
X = XLSX.readxlsx("влажность.xlsx")
лист = X[1]
влажность = Float64.(лист["A"][2:end])
println("Number of values: ", length(влажность),)
println("Range: ", minimum(влажность), " ... ", maximum(влажность), " %")
println("The first 5 values: ", влажность[1:5])
Data Set Transformation
We transform a data set when some of the original columns are not needed, but new data from external sources needs to be added. Delete the NSM, WeekStatus, Day_of_week, and Load_Type columns, and then add the temperature and humidity columns that were loaded earlier.
select!(df_energy, Not([:NSM, :WeekStatus, :Day_of_week, :Load_Type]))
println("Столбцы после удаления: \n\n", names(df_energy))
Add previously loaded temperature and humidity variables as new columns. The number of rows must match.
df_energy.Temperature_C = температура
df_energy.Humidity_Pct = влажность
Let's display the first 10 rows of the updated dataset.
println("The first 10 lines:")
first(df_energy, 10)
Save the converted data set to a new CSV file. The source file Steel_Industry.csv it remains unchanged - all transformations are saved in a new file. Сталелитейные_данные.csv.
CSV.write("Сталелитейные_данные.csv", df_energy)
Converting a MAT file
The data structure in a MAT file can also be represented in different ways. Suppose we have received 4 bearing data files from the test bench:
- H_1_0.mat – fully functional bearing;
- B_11_1.mat – malfunction of the roller element;
- I_1_1.mat – inner ring malfunction;
- O_6_1.mat – malfunction of the outer ring.
But the data analysis model requires a different structure – one MAT file with four keys "normal", "roller", "inner", "outer", where:
normal – fully functional bearing;
roller – malfunction of the roller element;
inner – malfunction of the inner ring;
outer – malfunction of the outer ring.
Let's upload the first file.
норм_тест = matread("H_1_0.mat")
This file has a different data structure than the required one. Let's display the data.
норм_тест = норм_тест["H_1_0"]
The file has one key "H_1_0, the data in which is a matrix. We know that the acoustic data is located in the second column. We don't need any other data. Let's extract this data.
норм_тест = норм_тест[:,2]
Let's open the rest of the MAT files.
ролик_тест = matread("B_11_1.mat")
внут_тест = matread("I_1_1.mat")
внеш_тест = matread("O_6_1.mat")
Let's extract the matrices.
ролик_тест = ролик_тест["B_11_1"]
внут_тест = внут_тест["I_1_1"]
внеш_тест = внеш_тест["O_6_1"]
Next, we will extract the second column from each matrix in the same way.
ролик_тест = ролик_тест[:,2]
внут_тест = внут_тест[:,2]
внеш_тест = внеш_тест[:,2]
Now we have four vectors with acoustic data.: норм_тест ролик_тест, внут_тест, внеш_тест. They must be assembled into a single MAT file of the previously designated structure. Let's create a dictionary with the necessary keys.
данные = Dict(
"normal" => норм_тест,
"roller" => ролик_тест,
"inner" => внут_тест,
"outer" => внеш_тест)
Save this data to a new MAT file.
matwrite("Подшипник_тест.mat", данные)
Let's read the created file.
matread("Подшипник_тест.mat")
The structure of the created file corresponds to the required one. The data is ready for further use.
Conclusion
In the course of this work, the entire path of preparing engineering data and signals for analysis is considered. It shows how to uniformly load information from heterogeneous sources – tabular (CSV, XLSX), text, binary (MAT), audio (WAV) and graphic formats, and how to bring them into a form suitable for calculations: vectors, matrices and DataFrame tables. The operations of data structure control, timestamp transformation, time series visualization, spectral analysis via FFT, and signal compression to a compact set of informative features are demonstrated.
Special attention is paid to the transformation of data sets: combining tables from external sources, removing redundant columns, and repacking disparate MAT files into a single structure with specified keys. These operations reflect typical engineering practice tasks where the data storage format does not match the format required by the model or analysis methodology. The presented techniques form the basis for subsequent stages – statistical processing, building diagnostic models and machine learning.