Contemporary Problems of Social Work


Using prediction in filtering data for solving model tasks

Автор/Author: Gdanskiy N.I., Кarpov А.М., Коmova P.Y.

Скачать/Download | Посмотреть онлайн/View online

Список литературы/References:

1. Gdanskiy N. I., Karpov A. V., Samoilov V. G. Control of systems with two degrees of freedom

based on reduced-load prediction // Chemical and Petroleum Engineering, 2011, №7, P. 12–


90 91


2. McMillan G., Weiner S. The predictive control models - past, present and future, part 1 //

Automation and IT in the energy sector, №7, 2013, P. 26-29

3. Whittaker, E.T; Robinson, G (1924). The Calculus Of Observations. Blackie & Son. pp. 291–

6. OCLC 1187948.. “Graduation Formulae obtained by fitting a Polynomial.”

4. Guest, P.G. (2012) [1961]. “Ch. 7: Estimation of Polynomial Coefficients”. Numerical Methods

of Curve Fitting. Cambridge University Press. pp. 147–. ISBN 978-1-107-64695-7.

5. Savitzky, A.; Golay, M.J.E. (1964). “Smoothing and Differentiation of Data by Simplified Least

Squares Procedures”. Analytical Chemistry 36 (8): 1627–39. doi:10.1021/ac60214a047.

6. Savitzky, Abraham (1989). “A Historic Collaboration”. Analytical Chemistry 61 (15): 921A–

3A. doi:10.1021/ac00190a744.

7. Steinier, Jean; Termonia, Yves; Deltour, Jules (1972). “Smoothing and differentiation of data

by simplified least square procedure”. Analytical Chemistry 44 (11): 1906–9. doi:10.1021/


8. Larive, Cynthia K.; Sweedler, Jonathan V. (2013). “Celebrating the 75th Anniversary of the

ACS Division of Analytical Chemistry: A Special Collection of the Most Highly Cited Analytical

Chemistry Papers Published between 1938 and 2012”. Analytical Chemistry 85 (0): 4201–2.


9. Riordon, James; Zubritsky, Elizabeth; Newman, Alan (2000). “Top 10 Articles”. Analytical

Chemistry 72 (9): 24 A–329 A. doi:10.1021/ac002801q.

10. Hydrologic Variability of the Cosumnes River Floodplain(Booth et al., San Francisco Estuary

and Watershed Science, Volume 4, Issue 2, 2006)

11. Statistical Analysis, Ya-lun Chou, Holt International, 1975,ISBN 0-03-089422-0, section 17.9.

12. The derivation and properties of the simple central moving average are given in full

at Savitzky–Golay filter


14. Weighted Moving Averages: The Basics”. Investopedia.


16. NIST/SEMATECH e-Handbook of Statistical Methods: Single Exponential Smoothing at

the National Institute of Standards and Technology

17. NIST/SEMATECH e-Handbook of Statistical Methods: EWMA Control Charts at the National

Institute of Standards and Technology geometric series



References in Roman script:

1. Gdanskiy N. I., Karpov A. v., Samoylov v. g. Upravlenie sistemami s dvumya stepenyami

svobody na osnove snizhennoy nagruzki prognozirovaniya // himicheskoe i neftegazovoe

mashinostroenie, 2011, №7, s. 12-15

2. Makmillan g., Vayner S. o prognosticheskoy modeli upravleniya - proshloe, nastoyashee i

budushee, chast’ 1 // Avtomatizatsiya i IT v energetike, №7, 2013, str. 26-29

3. Uitteker, E. T.; Robinson, G (1924). Ischislenie Nablyudeniy. Bleki I Syn. s. 291-6. Tsentr oclc

1187948.. “vypusknoy Formuly, poluchennye putem approksimatsii Polinomom.”

4. Gost’, P. G. (2012) [1961]. “SN. 7: otsenki Polinomial’nyh koeffitsientov”. Chislennye metody

approksimatsii Krivoy. Kembridzh Yuniversiti Press. s. 147–. Nomer ISBN 978-1-107-64695-7.

5. Savitskiy, A.; Goleya, M. Zh. E. (1964). “Sglazhivaniya i differentsirovaniya dannyh putem

uproshennoy protsedury naimen’shih kvadratov”. Analiticheskaya Himiya 36 (8): 1627-39.


6. Savitskiy, Avraam (1989). “Istoricheskoe Sotrudnichestvo”. Analiticheskaya Himiya 61 (15):

921A–3A. doi:10.1021/ac00190a744.

7. Steinier, Zhan; Termonia, Iv; Otel’ Deltour, Zhyul’ (1972). “Sglazhivaniya i differentsirovaniya

dannyh po uproshennoy protsedure naimen’shih kvadratov”. Analiticheskaya Himiya 44 (11):

1906-9. doi:10.1021/ac60319a045.

8. Lariv, Sintiya K.; Sweedler, Dzhonatan V. (2013). “Prazdnovanie 75-letiya ASU Otdel

analiticheskoy himii: spetsial’naya Kollektsiya samyh vysoko tsitiruemyh statey,

opublikovannyh analiticheskoy himii mezhdu 1938 i 2012 godov”. Analiticheskaya Himiya

85 (0): 4201-2.doi:10.1021/ac401048d.

9. Riordon, Dzheyms; Zubritskiy, Elizabet N’yuman, Alan (2000). “Top 10 Statey”. Analiticheskaya

Himiya 72 (9): 24 V–329 A. doi:10.1021/ac002801q.

10. Gidrologicheskaya izmenchivost’ Cosumnes Poyma reki(But i soavt., San-Frantsisko ust’e reki

i vodorazdel nauki, tom 4, Vypusk 2, 2006)

11. Statisticheskiy analiz, Ya-lun Chzhou, Holt interneshenel, 1975,s. ISBN 0-03-089422-0, razdel


12. Poluchenie i svoystva prostyh tsentral’noy skol’zyashey sredney privedeny v polnom ob#eme

na Savitskogo–Goleya fil’tr


14. Vzveshennyh Skol’zyashih Srednih: Osnovy”. Vashi lichnye.


16. V nist/SEMATECH elektronnyy Spravochnik statisticheskih metodov: odnokratnoe

Eksponentsial’noe sglazhivanie v Natsional’nom Institute standartov i tehnologii

17. V nist/SEMATECH elektronnyy Spravochnik statisticheskih metodov: kontrol’nye diagrammy

EWMA v Natsional’nom Institute standartov i tehnologii Geometricheskaya seriya



Содержание статьи/Article:

The article considers the actual problem of digital control algorithms accuracy increase.

In particular, it is necessary to filter digital data received via communication with peripheral

sources.It is necessary to improve methods of digital control [1-3].

Digital techniques compared to analogous allow to use more complex control algorithms to

eliminate the dependence on temperature, etc. However, it is necessary to provide a number of

conditions for the practical realization of these advantages. The most important among them are

the limits on the performed calculations and the accuracy of the calculated control parameters.

The first step in construction control scheme is the processing of feedback data. The digital

information which is transmitted via high-speed communication ports, always contains errors

due to the measurement process, digitization, noise and other factors.

One of the main problems is to clear input from the noise. The adequacy and accuracy of

control action depends on the successful solution of this problem.

Before turning to our algorithm will consider the most popular ones.

One of the most popular methods is moving average [19]. In statistics, a moving

average (rolling average or running average) is a calculation to analyze data points by

creating a series of averages of different subsets of the full data set. It is also called a moving

mean (MM) [10] or rolling mean and is a type of finite impulse response filter. Variations

include: simple, and cumulative, or weighted forms (described below).

Given a series of numbers and a fixed subset size, the first element of the moving average

is obtained by taking the average of the initial fixed subset of the number series. Then the

subset is modified by “shifting forward”; that is, excluding the first number of the series and

including the next number following the original subset in the series. This creates a new subset

82 83


of numbers, which is averaged. This process is repeated over the entire data series. The plot line

connecting all the (fixed) averages is the moving average. A moving average is a set of numbers,

each of which is the average of the corresponding subset of a larger set of datum points. A

moving average may also use unequal weights for each datum value in the subset to emphasize

particular values in the subset.

A moving average is commonly used with time series data to smooth out short-term

fluctuations and highlight longer-term trends or cycles. The threshold between short-term

and long-term depends on the application, and the parameters of the moving average will

be set accordingly. For example, it is often used in technical analysis of financial data, like

stock prices, returns or trading volumes. It is also used in economics to examine gross domestic

product, employment or other macroeconomic time series. Mathematically, a moving average is

a type of convolution and so it can be viewed as an example of a low-pass filter used in signal

processing. When used with non-time series data, a moving average filters higher frequency

components without any specific connection to time, although typically some kind of ordering

is implied. Viewed simplistically it can be regarded as smoothing the data.

In financial applications a simple moving average (SMA) is the unweighted mean of the

previous data. However, in science and engineering the mean is normally taken from an equal

number of data on either side of a central value. This ensures that variations in the mean are

aligned with the variations in the data rather than being shifted in time.

The period selected depends on the type of movement of interest, such as short, intermediate,

or long-term. In financial terms moving-average levels can be interpreted as support in a rising

market, or resistance in a falling market.

If the data used are not centered around the mean, a simple moving average lags behind the

latest datum point by half the sample width. An SMA can also be disproportionately influenced

by old datum points dropping out or new data coming in. One characteristic of the SMA is that

if the data have a periodic fluctuation, then applying an SMA of that period will eliminate that

variation (the average always containing one complete cycle). But a perfectly regular cycle is

rarely encountered.[11]

For a number of applications, it is advantageous to avoid the shifting induced by using only

‘past’ data. Hence a central moving average can be computed, using data equally spaced on

either side of the point in the series where the mean is calculated.[12] This requires using an odd

number of datum points in the sample window.

A major drawback of the SMA is that it lets through a significant amount of the signal shorter

than the window length. Worse, it actually inverts it. This can lead to unexpected artifacts, such

as peaks in the smoothed result appearing where there were troughs in the data. It also leads

to the result being less smooth than expected since some of the higher frequencies are not

properly removed.

The problem can be overcome by iterating the process three times, with the window being

shortened by a factor of 1.4303 at each step.[13] This removes the negation effects and provides

a better behaved filter. This solution is often used in real-time audio filtering since it is

computationally quicker than other comparable filters such as a gaussian kernel.

In a cumulative moving average, the data arrive in an ordered datum stream, and the user

would like to get the average of all of the data up until the current datum point.

The brute-force method to calculate this would be to store all of the data and calculate the

sum and divide by the number of datum points every time a new datum point arrived. However,

it is possible to simply update cumulative average as a new value, becomes available, using

the formula:

Thus the current cumulative average for a new datum point is equal to the previous cumulative

average, times , plus the latest datum point, all divided by the number of points received so far,

. When all of the datum points arrive ( ), then the cumulative average will equal

the final average.

The derivation of the cumulative average formula is straightforward. Using

and similarly for , it is seen that

Solving this equation for results in:

A weighted average is any average that has multiplying factors to give different weights

to data at different positions in the sample window. Mathematically, the moving average is

the convolution of the datum points with a fixed weighting function. One application is

removing pixelisation from a digital graphical image.

In technical analysis of financial data, a weighted moving average (WMA) has the specific

meaning of weights that decrease in arithmetical progression.[14] In an -day WMA the latest

day has weight , the second latest , etc., down to one.

An exponential moving average (EMA), also known as an exponentially weighted moving

average (EWMA),[15] is a type of infinite impulse response filter that applies weighting factors

which decrease exponentially. The weighting for each older datum decreases exponentially,

never reaching zero. The graph at right shows an example of the weight decrease.

The EMA for a series may be calculated recursively:

For ,


• The coefficient represents the degree of weighting decrease, a constant smoothing

factor between and . A higher discounts older observations faster.

• is the value at a time period .

• is the value of the EMA at any time period .

is undefined. may be initialized in a number of different ways, most commonly by

setting to , though other techniques exist, such as setting to an average of the first

or observations. The importance of the initialisations effect on the resultant moving

average depends on ; smaller values make the choice of relatively more important than

larger values, since a higher α discounts older observations faster.

Whatever is done for it assumes something about values prior to the available data and

is necessarily in error. In view of this the early results should be regarded as unreliable until

the iterations have had time to converge. This is sometimes called a ‘spin-up’ interval. One way

to assess when it can be regarded as reliable is to consider the required accuracy of the result.

For example, if accuracy is required, initialising with and taking data after five time

84 85


constants (defined above) will ensure that the calculation has converged to within (only

< of will remain in the result ). Sometimes with very small alpha, this can mean little of

the result is useful. This is analogous to the problem of using a convolution filter (such as a

weighted average) with a very long window.

This formulation is according to Hunter (1986).[16] By repeated application of this formula

for different times, we can eventually write St as a weighted sum of the datum points , as:

for any suitable The weight of the general datum point is .

An alternate approach by Roberts (1959) uses in lieu of :[17]

From a statistical point of view, the moving average, when used to estimate the underlying

trend in a time series, is susceptible to rare events such as rapid shocks or other anomalies.

A Savitzky–Golay filter[20] is a digital filter that can be applied to a set of digital data points

for the purpose of smoothing the data, that is, to increase the signal-to-noise ratio without

greatly distorting the signal. This is achieved, in a process known as convolution, by fitting

successive sub-sets of adjacent data points with a low-degree polynomial by the method

of linear least squares . When the data points are equally spaced an analytical solution to the

least-squares equations can be found, in the form of a single set of “convolution coefficients”

that can be applied to all data sub-sets, to give estimates of the smoothed signal, (or derivatives

of the smoothed signal) at the central point of each sub-set. The method, based on established

mathematical procedures,[3][4] was popularized by Abraham Savitzky and Marcel J. E.

Golay who published tables of convolution coefficients for various polynomials and sub-set

sizes in 1964.[5][6] Some errors in the tables have been corrected.[7]The method has been

extended for the treatment of 2- and 3-dimensional data.

Savitzky and Golay’s paper is one of the most widely cited papers in the journal Analytical

Chemistry[8] and is classed by that journal as one of its “10 seminal papers” saying “it can be

argued that the dawn of the computer-controlled analytical instrument can be traced to this








0 2 4 6 8 10 12

Initial data

Method moving average

Method moving average with


Figure 1. Data with outlier and moving average method

We can see that all these methods are not good when we have data with even single outlier.

So let us describe a new one.

Let be the value of discrete values that are transmitted through feedback channels.

is obtained in the measurement of time-varying electrical and mechanical parameters

that are inert by nature (for example, the angle of shaft rotation, and the power supply voltage,

etc.). This data comes in discrete time .

Suppose that in a neighborhood of the current node times

are set. The measured values are given in these nodes. We pose

the problem in the following form: it is necessary to filter the measured values of the function

while saving useful information.

The principle of reduction of errors is the main disadvantage of most methods. Even large

errors are not eliminated entirely, but are taken into account in the neighboring nodes to match

the selected numerical criterion of optimality.







0 2 4 6 8 10 12

Initial data

Method moving average

Method moving average with


Figure 2. Approximations results using standard methods.

To approximate the function we will use polynomials of degree :


We construct the approximating polynomial of a set of points in a certain neighborhood,

using the method of least squares.

Use neighborhood that includes nodes for the

original version of the polynomial. At each step we filter data in . We assume that

we have already applied the filter in butuse unfiltered data for the desired range,

because the use of data already filtered for further filtering may lead to accumulated error.

This neighborhood may contain outliers. Therefore we eliminate nodes , in which measured

values are very different from the predicted: . Let us find using the

linear approximation values in the neighboring nodes. , and

is standard deviation. Suppose m is equal to . So the resulting formula for the emission is:

86 87



Note that condition (2) can be used only for internal points of the original range. That is, we

check points with indices 2, ..., N-1 to filter array 1, ..., N.

Such method of pre-filtering data can eliminate single outlier. However, in practice, there

are more complicated situations. For example, consecutive outliers of two or more points. We

propose to use the median filter to eliminate these outliers. We consider it in relation to our


In signal processing, it is often desirable to be able to perform some kind of noise reduction on

a signal. The median filter is a nonlinear digital filtering technique, often used to remove noise.

Such noise reduction is a typical pre-processing step to improve the results of later processing.

The main idea of the median filter is to run through the signal entry by entry, replacing each

entry with the median of neighboring entries. The pattern of neighbors is called the “window”,

which slides, entry by entry, over the entire signal. The most obvious window is just the first few

preceding and following entries. Note that if the window has an odd number of entries, then

the median is simple to define: it is just the middle value after all the entries in the window are

sorted numerically.

Because there is no entry preceding the first value, , and are used to

calculate median for .

We take the following steps to find the median of a sequence of points:

1. Sort the points in ascending order.

2. If the number is odd ( ), then the median is the average value of sorted

points ( ). If is even ( ), the median is the average of the values of two points

located in the middle (with indices and ).

It is necessary to reduce the order of the median, if there is not enough data to calculate it.

For processing end-points values , and are used.

Use the following algorithm for data preprocessing. We calculate the median value for each

of the points with a certain parameter . Compare the value at each point with the value of its

median. If the absolute value of their difference is greater than , then a point is considered

outlier. Replace its value with the median value. Next, use the method of least squares.

Let the degree of the polynomial be equal to three. This will reduce the amount of

computation and give a good result. Accuracy of approximation will improve very slowly with

polynomial degree increase. In most cases, the result will be visually worse.

Parameter is called the half-width of the base neighborhood filtered data. Half-width should

be selected on the basis of specific subject area in which the results could be approximated by a

polynomial with a selected degree. To estimate the value of the half-width use a specific

frequency of received data and a suitable time range.

Let be coefficient of proximity to the experimental data:

. If then assuming a normal distribution of noise,

not more than 20% will not satisfy this condition.

Considered mathematical relationships depend on the following parameters:

1. is the error scale,

2. is the coefficient of proximity to the experimental data

3. is the degree approximating polynomial

4. is the half-width of the base neighborhood.

5. is the window for median filter.

The parameter can be set based on the standard deviation ( ) and adjust during

the program with real data.

Take the parameter k equal to 1.5, based on the assumption that the noise has a normal

distribution. It is proposed to maintain constant during the entire execution of the algorithm.

Approximating polynomial P_s (t) degree is assumed equal to three (s = 3).

The parameter h will change based on the different frequency model (or experimental) data.

We propose the following general algorithm


1. is the array with time moments.

2. is the array with noise.

3. is the scale nodal deviations function.

4. is the coefficient of proximity to the experimental data

5. is the degree approximating polynomial

6. is the half-width of the base neighborhood.

7. is the pattern of neighbors

Statement of the problem.

The task is to filter the values of f (t) for a given time t. The filter data is .

Additional array.

1) is boolean array with outline points .


Step 1. Initial Actions.

1.1. Assigning values to constant parameters of the algorithm: k: = 1.5.

1.2. Calculate or set the parameter for a given task or the first results obtained.

1.3. Set the initial values of the array B:

1.4. For noise reduction preprocessing step we set when

or use median filter 1.5. If number of outlier is greater

then 20% of the N then we should correct m and do 1.3 and 1.4 again.

1.5. Calculate median in each node . If then .

Step 2. Let . . is the last point in our window. While

do the following steps.

1.1. is the current pattern of neighbors we use in approximation.

1.2. Construct a polynomial using the method of least squares, taking the filtered

points or such points where .

1.3. Calculating new values in an array . If then

calculate .

1.4. Calculate the standard deviation . Correct the value .

1.5. Let . Repeat all the steps.

Completion of the algorithm.

Let’s test our algorithm on an example from the beginning of the article. Each of the chosen

methods attempts to smooth the error from a single point. Our algorithm will remove the error

completely. Thus, the standard deviation from the actual data becomes zero (Figure 3.). We

used the following parameters:

1. is the scale nodal deviations function.

2. is the coefficient of proximity to the experimental data

3. is the degree approximating polynomial

4. is the half-width of the base neighborhood.

5. is the pattern of neighbors

88 89


Figure 3. The result of the filtering algorithm.

Consider the result of filtering the example of noisy data. Let us take in

using 100 points with standard deviation and outliers in several points. Our method

not only removed the outliers, but also made a good approximation to the value of the original

function by removing noise. We used the following parameters:

1. is the scale nodal deviations function.

2. is the coefficient of proximity to the experimental data

3. is the degree approximating polynomial

4. is the half-width of the base neighborhood.

5. is the pattern of neighbors

Figure 4. The result of the filtering algorithm with noise and outliers

Let us take in using 1000 points with standard deviation and

outliers in several points. Our method not only removed the outliers, but also made a good

approximation to the value of the original function by removing noise. We used the following


6. is the scale nodal deviations function.

7. is the coefficient of proximity to the experimental data

8. is the degree approximating polynomial

9. is the half-width of the base neighborhood.

10. is the pattern of neighbors

Figure 5. The result of the filtering algorithm with noise and outliers

The Results.

The paper established a new principle of filtering digital data. This method not only effectively

removes outliers, but also makes high-quality anti-aliasing, removing noise. As a result, based

on the received communication lines digitized data computed real values of measured physical

quantities, which are much closer to their true values. The program was written that implements

the algorithm. Efficiency and effectiveness of the algorithm is demonstrated by examples.

Introduced a new approach to input filtering in digital control systems can significantly

increase the accuracy by eliminating them from significant errors. This approach will allow, in

particular, to a considerable extent reduce the digital data error caused by their conversion to

numerical form.

Ключевые слова/Tags1: Digital control, feedback data, data filtering, least squares method