Engineering analysis of telemetry data of a steel mill
Introduction
Every industrial enterprise generates huge amounts of data every day: meter readings, process parameters, weather reports, and emissions data. The engineering value of this data appears at the moment when we look for patterns in them.: how the parameters are related, what operating modes exist, and whether there are hidden optimization cycles.
In this paper, we will consistently analyze the telemetry data of the steel company for 2018. The set contains 35,040 entries with a 15-minute interval: active and reactive energy consumption, CO₂ emissions, power factor, temperature and humidity. We will use all the tools of an engineering analyst: descriptive statistics, correlations, visualization, division into seasons and loading levels, the principal component method, clustering and spectral analysis. The goal is not just to build graphs, but to understand how the company operates and where the efficiency reserves are hidden.
Each step is accompanied by an engineering interpretation. We will see that the high correlation between consumption and CO₂ emissions allows us to indirectly control environmental indicators. We will find three stable modes of operation with different efficiencies. We will find the daily and weekly cycles that are important in load planning. Power factor analysis will show where energy is being wasted. As a result, we will get a complete picture of the production system, suitable for making engineering decisions.
Connecting libraries
We will connect the libraries necessary for research data analysis. Each of them solves its own problem:
-
Statistics and StatsBase – basic statistical functions: mean, standard deviation, quantiles, correlations;
-
StatsPlots and Plots – plotting: histograms, scatter plots, heat maps, box diagrams;
-
CSV and DataFrames – reading and processing tabular data;
-
Dates – working with dates and times, which is important when analyzing time series;
-
Clustering – clustering algorithms, in particular k-means;
-
MultivariateStats – Principal component method for dimensionality reduction;
-
HypothesisTests – statistical criteria for hypothesis testing;
-
FFTW – Fast Fourier transform for spectral analysis;
-
Random – reproducibility of results for random operations.
for pkg in (["Statistics", "StatsBase", "StatsPlots", "CSV", "DataFrames", "Dates", "Clustering", "MultivariateStats", "HypothesisTests", "FFTW", "Random"])
try
eval(Meta.parse("using $pkg"))
catch
Pkg.add(pkg)
eval(Meta.parse("using $pkg"))
end
end
Uploading a data set
Let's upload a CSV file with the telemetry data of the steel company. It contains 35040 records with energy consumption, temperature and humidity data for 2018.
df = CSV.read("Сталелитейные_данные.csv", DataFrame)
println("Dimension: ", size(df))
println("Columns: ", names(df))
Date Conversion
Create a DateTime column for time analysis.
df.DateTime = DateTime.(df.date, dateformat"dd/mm/yyyy HH:MM")
println("Range: ", minimum(df.DateTime), " - ", maximum(df.DateTime))
Descriptive statistics
The first step of any analysis is to get a summary of all numeric variables. Function describe It will show the average, minimum, maximum, standard deviation and quartiles. This allows you to quickly estimate the variation of parameters and detect anomalies.
числовые = [:Usage_kWh, :Lagging_Current_Reactive_Power_kVarh,
:Leading_Current_Reactive_Power_kVarh, :CO2_tCO2_,
:Lagging_Current_Power_Factor, :Leading_Current_Power_Factor,
:Temperature_C, :Humidity_Pct]
describe(df[:, числовые], :all)
The correlation matrix
Let's build a heat map of correlations between all numerical parameters. It is a key research analysis tool.: we can immediately see which parameters are related to each other. For example, we expect a high correlation between energy consumption and CO₂ emissions.
corr_matrix = cor(Matrix(df[:, числовые]))
heatmap(corr_matrix,
xticks=(1:length(числовые), string.(числовые)),
yticks=(1:length(числовые), string.(числовые)),
xrotation=45, size=(700, 600),
title="Correlation matrix of parameters")
Numerical values of correlations
Let's output the numerical values of the correlations for a more accurate estimate. Let's sort the parameter pairs in descending order of the absolute correlation value.
println("Топ-10 корреляций:\n")
корреляции = Float64[]
пары = String[]
for i in 1:length(числовые)
for j in (i+1):length(числовые)
push!(корреляции, corr_matrix[i, j])
push!(пары, "$(числовые[i]) - $(числовые[j])")
end
end
idx = sortperm(abs.(корреляции), rev=true)
for k in 1:10
println(" $(пары[idx[k]]): $(round(корреляции[idx[k]], digits=3))")
end
Color-coded scattering graph
Let's construct a scattering diagram for the two most related parameters. Let's add color coding by temperature. This allows you to see how the third variable affects the relationship of the other two.
scatter(df.Usage_kWh[1:1000], df.CO2_tCO2_[1:1000],
marker_z=df.Temperature_C[1:1000],
xlabel="Energy consumption, kWh", ylabel="CO emissions",
title="Energy consumption and CO emissions",
color=:thermal, legend=:bottomright, size=(700, 500))
Seasonal analysis
Let's add a column with the season of the year based on the month. Then we compare the average consumption by season. This is a typical engineering analysis: break the data into groups and compare statistics.
df.Season = [["Зима", "Зима", "Весна", "Весна", "Весна", "Лето",
"Лето", "Лето", "Осень", "Осень", "Осень", "Зима"][month(dt)]
for dt in df.DateTime]
season_stats = combine(groupby(df, :Season),
:Usage_kWh => mean => :avg_usage,
:Usage_kWh => std => :std_usage,
:Temperature_C => mean => :avg_temp)
println("Statistics by season:")
season_stats
Visualization by season
Let's build a diagram (boxplot) of energy consumption by season. The chart shows the median, quartiles, and outliers-this gives a more complete picture of the distribution than just the average.
boxplot(df.Season, df.Usage_kWh,
xlabel="Season", ylabel="Energy consumption, kWh",
title="Distribution of energy consumption by season",
legend=false, size=(700, 450))
Time of day analysis
Let's extract the hour of the day from the DateTime and analyze the daily consumption profile. This is important for understanding the operating modes of the enterprise.
df.Hour = hour.(df.DateTime)
hourly_stats = combine(groupby(df, :Hour),
:Usage_kWh => mean => :avg_usage,
:Usage_kWh => std => :std_usage)
plot(hourly_stats.Hour, hourly_stats.avg_usage,
ribbon=hourly_stats.std_usage,
xlabel="The hour of the day", ylabel="Average consumption, kWh",
title="Daily energy consumption profile",
legend=false, linewidth=2, size=(700, 400))
Adding download features
We use data on energy consumption to classify the load level. We will divide it into three quartile categories: low, medium and high load.
q33, q67 = quantile(df.Usage_kWh, [0.33, 0.67])
df.LoadLevel = [x <= q33 ? "Низкая" : x <= q67 ? "Средняя" : "Высокая"
for x in df.Usage_kWh]
println("Границы: низкая < $(round(q33, digits=1)), средняя < $(round(q67, digits=1)), высокая")
println("Distribution by levels:")
combine(groupby(df, :LoadLevel), nrow => :count)
Multivariate Analysis: PCA
Let's apply the Principal Component Analysis method to reduce dimensionality. This will allow you to see the data structure in two-dimensional space and discover hidden patterns. PCA finds the directions of maximum data spread.
pca_data = Matrix(df[:, числовые])
pca_data = (pca_data .- mean(pca_data, dims=1)) ./ std(pca_data, dims=1)
pca_model = fit(PCA, pca_data', maxoutdim=2)
проекция = MultivariateStats.transform(pca_model, pca_data')
общая_var = tvar(pca_model)
объясн_var = principalvar(pca_model, 2)
println("Explained variance: ", round.(объясн_var / общая_var, digits=3))
scatter(проекция[1, 1:2000], проекция[2, 1:2000],
xlabel="PC1", ylabel="PC2",
title="Energy consumption PCA (first 2000 points)",
alpha=0.5, legend=false, size=(600, 450))
Clustering of operating modes
Let's apply the k-means algorithm to identify typical operating modes of an enterprise. This is a method for automatically grouping data without a teacher. Let's try to identify 3 clusters and see if they match the load levels.
k = 3
cluster_result = kmeans(pca_data', k)
df.Cluster = cluster_result.assignments
println("Размеры кластеров:\n")
for i in 1:k
println(" Кластер $i: $(sum(df.Cluster .== i)) точек")
end
Visualization of clusters
Let's display the clusters on the PCA projection in different colors. This will show how well the algorithm has separated the data.
scatter(проекция[1, 1:2000], проекция[2, 1:2000],
group=df.Cluster[1:2000],
xlabel="PC1", ylabel="PC2",
title="Clusters of operating modes",
legend=false,
alpha=0.5, size=(600, 450))
Characteristics of clusters
For each cluster, we calculate the average values of the key parameters. This will make it possible to interpret clusters: for example, a cluster with high energy consumption and high temperature may correspond to the day shift in summer.
cluster_stats = combine(groupby(df, :Cluster),
:Usage_kWh => mean => :avg_usage,
:Temperature_C => mean => :avg_temp,
:Humidity_Pct => mean => :avg_humidity,
:Lagging_Current_Power_Factor => mean => :avg_pf)
println("Characteristics of clusters:")
cluster_stats
Hypothesis testing: weekdays and weekends
Let's add a day of the week attribute and test the statistical hypothesis: whether the energy consumption differs on weekdays and weekends. We use the Student's t-test for independent samples.
df.DayOfWeek = dayofweek.(df.DateTime)
df.IsWeekend = df.DayOfWeek .>= 6
будни = df.Usage_kWh[.!df.IsWeekend]
выходные = df.Usage_kWh[df.IsWeekend]
println("Average on weekdays: ", round(mean(будни), digits=2))
println("Average on weekends: ", round(mean(выходные), digits=2))
println("The t-test: ", UnequalVarianceTTest(будни, выходные))
Reactive power and power factor
Let's analyze the power factor, an important indicator of energy efficiency. A low power factor means that the company consumes a lot of reactive energy, which can incur fines.
histogram(df.Lagging_Current_Power_Factor, bins=40,
xlabel="Power factor, %", ylabel="Frequency",
title="Power factor distribution",
legend=false, size=(600, 400))
vline!([mean(df.Lagging_Current_Power_Factor)],
linewidth=2, linestyle=:dash, label="Average")
Spectral analysis of the time series
Let's apply the FFT to the consumption time series to find the periodic components. This will help identify daily, weekly, and seasonal cycles.
signal = df.Usage_kWh[1:4096] .- mean(df.Usage_kWh[1:4096])
spectrum = abs.(fft(signal))
freqs = (0:2047) * (1/0.25) / 4096 * 24 # cycles per day
plot(freqs[2:100], spectrum[2:100],
xlabel="Частота, циклов/сутки", ylabel="The amplitude",
title="The range of energy consumption",
linewidth=1.5, legend=false, size=(700, 400))
vline!([1.0], linestyle=:dash, label="Daily cycle")
vline!([1/7], linestyle=:dot, label="Weekly cycle")
Saving the analysis results
Save the processed DataFrame with all the added attributes (season, hour, load level, cluster) to a new CSV file.
CSV.write("Новые_данные.csv", df)
println("Columns: ", names(df))
Conclusion
We conducted an end-to-end research analysis of the steel company's telemetry data, going from the initial table to meaningful engineering conclusions. The data contains a rich internal structure, and we have learned how to extract it sequentially.
Correlation analysis revealed the strongest relationship between energy consumption and CO₂ emissions, which allows indirect environmental monitoring based on meter data. Seasonal analysis showed an increase in consumption in winter due to the heating load. The daily profile revealed a two-peak regime with a morning rise and an evening decline. The division into loading levels yielded three uniform groups of observations.
The principal component method showed the data structure in a two-dimensional space, and clustering identified three stable modes: high load, low load with poor efficiency, and low load with excellent efficiency. Spectral analysis confirmed the daily and weekly cycles, as well as the two-shift schedule. The histogram of the power factor with two peaks completed the picture.
All the conclusions were obtained without complex machine learning models, only statistics and visualization. The obtained patterns can become the basis for the selection of features, the formulation of forecasting tasks and the development of monitoring systems.