The Degree of Best Downsampled Convolutional Neural Network Approximation in Terms of Orlicz Modulus of Smoothness

Orlicz spaces are one of the essential wide spaces that could help in practical usage. It is interesting to study approximation of convolutional neural networks (CNNs) in Orlicz spaces, to get degree of approximation in terms of modulus of smoothness. We define an extended class of Orlicz function, called quasi-Orlicz space.

We introduce the introduction within two sections. The first section is about the space of interest, that is Orlicz space. It includes preliminaries and development stations about it. Then we define Orlicz space and modulus of smoothness with the auxiliary results about them. Second, we introduce the literature review about CNNs and sampling method. So we construct our CNN to get theoretical approximation results in terms of Orlicz modulus of smoothness. Then an application about approximation of CNNs is introduced to get best approximation for function from generalized Orlicz space.

1.1 Orlicz spaces

The first who studied this type of spaces was the mathematician Orlicz [1]. As a generalization of the Lebesgue integrable spaces ${{L}_{p}}$. With completeness property, Orlicz space form an extended Banach space:

${{L}_{\Phi }}\left( \mu  \right)=\left\{ f:f~\text{is }\!\!~\!\!\text{ }\mu -\text{meas }\!\!~\!\!\text{ and }\!\!~\!\!\text{ }||{{f}||_{\Phi }}<\infty  \right\}$                 (1)

where, the norm ${{||·||}_{\Phi }}$ had been defined in many ways to cover Orlicz space, beginning with Orlicz him self with his norm.

$||f||_{\Phi }^{0}=\sup \left\{ \underset{T}{\mathop \int }\,\left| f\left( t \right)y\left( t \right) \right|d\mu ~:y\in {{L}_{\Psi ,}}{{I}_{\Psi }}\left( y \right)\le 1 \right\}$                 (2)

where, Orlicz norm was defined depending on Young function $\Phi $ and the complementary function $\Psi $, the generator of the modular unit ball. The function $\Psi $ is defined by

$\Psi \left( u \right)=\text{sup}\left\{ \mid u\mid v-\Phi \left( v \right)\,\!:v\ge 0 \right\}$                   (3)

for all $u\in \mathbb{R}$.

In simultaneous times in the fifteenth, Nakano [2], Morse and Transue [3] and Luxembourg [4] investigated the Luxembourg norm, which has been defined, using the concept of functional Minikowski over a convex modular unit ball, $\left\{ x:~I\left( x \right)\le 1 \right\}$, with

${||{f}||_{\Phi }}=\underset{\lambda >0}{\mathop{\text{inf}}}\,\left\{ {{I}_{\Phi }}\left( \frac{f}{\lambda } \right)\le 1 \right\}$                    (4)

Another main norm was defined by Amemiya at the same time, named after his name $p$-Amemiya norm [2].

$||f||_{\Phi }^{A}=\underset{\lambda >0}{\mathop{\inf }}\,\frac{1}{\lambda }\left( 1+{{I}_{\Phi }}\left( \lambda f \right) \right)$                  (5)

In separated papers, Krasnoselskii and RutickiI [5], Nakano [2] and Luxembourg and Zaneen [6] proved under additional conditions on $\Phi $, that Amemiy norm $||·||_{\Phi }^{A}$ is exactly the Orlicz norm$||·||_{\Phi }^{0}$. Moreover, Cui et al. [7] gave the basic results about the so called $p$-Amemiya norms equipped with other spaces. Also, Wisła [8] showed an overview of the developments of the spaces defined later. Moreover, Wisła [9] introduced a more generalized $p$-Amemiya type norms by restricting his conditions about the outer function. Later, he introduced the concept of outer function $s$ and get $s$-norms on Orlicz spaces.

${||{f}||_{\Phi ,s}}=\underset{\lambda >0~}{\mathop{\inf }}\,\frac{1}{\lambda }s\left( {{I}_{\Phi }}\left( \lambda f \right) \right)$                     (6)

Until today, many papers improved the norms of Orlicz [10-12].

All the above definitions of Orlicz norms depend on the values of $p$ to be within $\left[ 1,~\infty  \right]$. For the case$~0<p<1$, AL-Janabi and Almurieb [13] made another contribution to Orlicz spaces via outer function ${{S}_{p}}:\left[ 0,\infty  \right)\to \left[ 0,\infty  \right)$ in the following definition.

Definition 1. [13] For $0<p<1$ the family ${{S}_{p}}$ of functions is defined by

 ${{S}_{p}}~\left( f \right)={{\left( 1+{{f}^{2}} \right)}^{\frac{1}{p}}}$                 (7)

where, $f\in {{L}_{\Phi ,~{{S}_{p}}}}\left( {{I}^{d}} \right),$ the quasi-normed space.

${{L}_{\Phi ,{{S}_{p}}}}\left( {{I}^{d}} \right)=\left\{ f\in {{L}_{p}}\left( {{I}^{d}} \right)\left| {{||f||}_{\Phi ,~{{S}_{p}}}}<\infty  \right. \right\}$                 (8)

where,

${{||f||}_{\Phi ,{{S}_{p}}}}=\underset{\lambda >0}{\mathop{\inf }}\,\frac{1}{\lambda }{{\left( I_{\text{ }\!\!\Phi\!\!\text{ }}^{2}~\left( \lambda f \right)+1 \right)}^{\frac{1}{p}}}$                     (9)

As mentioned by AL-Janabi and Almurieb [13], ${{S}_{p}}$, $p<1$, are convex, strictly increasing and all ${{S}_{p}}$ match at zero. As concluded there, ${{L}_{\Phi ,~{{S}_{p}}}},~p\le 1$ quasi-normed space is a generalization of $p-$Amemiy’s normed space on $p\ge 1$.

Moreover, the degree of best approximation for functions from ${{L}_{\Phi ,~~{{S}_{p}}}}~$in terms of CNNs were studied [13], so that

$||f-{{N}||_{\Phi ,~~{{S}_{p}}}}<\epsilon $                 (10)

More precisely, the following theorem is given by AL-Janabi and Almurieb [13].

Theorem A. Let $f\in {{L}_{\text{ }\!\!\Phi\!\!\text{ },{{S}_{p}}}}\left( {{I}^{d}} \right)$,$~0<p<1$, then by (10) there exists $f_{j}^{w,b}$ s.t $||f-f{{_{J}^{w,b}}||_{\text{ }\!\!\Phi\!\!\text{ },{{S}_{p}}}}\le {{C}_{\left( p,k \right)}}{{||f||}_{\text{ }\!\!\Phi\!\!\text{ },{{S}_{p}}}}$.

Our target here is to improve the degree of approximation by studying modulus of smoothness in our Orlicz quasi-normed space in the following section.

1.2 Orlicz moduli of smoothness

Moduli of smoothness had been studied widely in Orlicz space. The importance of this comes from the need to improve the degree of function approximation via direct approaches. Direct approaches approximation implies a faster convergence to zero than general previous estimates. On the other hand, converse theorems give a characterization of smoothness of functions depending on its degree of approximation in direct approach. First results were given by Jackson [14], and Bernstein [15], for direct and inverse theorems, respectively, for the space of continuous functions in terms of modulus of continuous [16]. Later, second and third orders moduli of smoothness were involved in many papers concerning many wide generalizations of function’s spaces [17-22].

However, the birth of modulus of smoothness is much deeper in history [23-26], we concern here with moduli of smoothness as for as relates to Orlicz spaces. It’s very hard to investigate the early works of this topic, since they were written in Russian. However, the oldest paper we have obtained is attributed to Tsyganok [27] in 1966. He proved direct theorem as a modulus of continuity for Orlicz spaces. Later, in the eighteens of the last century, Ramazanov generalized direct theorem in Orlicz spaces for modulus of smoothness with higher orders [28]. In 1985, Musielak [24], generalized τ-modulus of smoothness, resulted by Popov [25], for Orlicz spaces. In 1991, Garidi generalized those results of Ramazanov [28], Garidi [29]. Many other results were studied by Israfilov and Akgün [30], Akgün [31], Chaichenko et al. [32], for weighted Orlicz spaces. More extensions were resulted to Orlicz spaces by Shidlich and Chaichenko [33, 34], which were combined by Chaichenko Shidlich et al. [32].

1.3 Modulus of Smoothness in ${{L}_{\Phi ,~{{S}_{p}}}}\left( {{I}^{d}} \right)$

Let’s now present the main definitions of our modulus of smoothness.

First, from Ditzian and Totik [23], define the following N-th symmetric difference

$\Delta_{h}^{N}\left( f\left( x \right) \right)=\underset{i=1}{\overset{N}{\mathop \sum }}\,{{\left( -1 \right)}^{i}}\left( \begin{matrix}   N  \\   i  \\ \end{matrix} \right)~f\left( x-ih \right)$                     (11)

The Orlicz Modulus of smoothness in terms of (11) is given by

${{\omega }_{N}}{{\left( f,\delta  \right)}_{\Phi ,~{{S}_{p}}}}=\underset{\left| h \right|\le \delta }{\mathop{\sup }}\,||\Delta_{h}^{N}{{\left( f \right)}||_{\Phi ,~{{S}_{p}}}}$                (12)

We study the main properties of the N-th symmetric difference (11) and the Orlicz modulus of smoothness (12) in the following section.

1.3.1 Modulus of smoothness properties (Auxiliary results)

In the following lemma, we prove that the $N-$th symmetric difference (11) satisfies the following properties under our Orlicz norm (9):

Lemma 1

Let $f~$be any function from (8), then

1. $||\Delta_{h}^{N}{{\left( f \right)}||_{\Phi ,~{{S}_{p}}}}\le k\left( N \right){{||f||}_{\Phi ,~{{S}_{p}}}}$, where $k\left( N \right)=\underset{i=0}{\overset{\infty }{\mathop \sum }}\,\left| \left( \begin{matrix}   N  \\   i  \\\end{matrix} \right) \right|\le {{2}^{N}},$ $N=\inf \left\{ k\in \mathbb{N}:k>N \right\}$.
2. $\left( \Delta_{h}^{N}\left( \Delta_{h}^{M}\left( f \right) \right) \right)\left( x \right)=\Delta _{h}^{N+M}\left( f\left( x \right) \right)\left( a.~e. \right)$.
3. $||\Delta_{h}^{N+M}{{\left( f \right)}||_{\Phi ,~{{S}_{p}}}}\le {{2}^{\left\{ M \right\}}}||\Delta_{h}^{N}{{\left( f \right)}||_{\Phi ,~{{S}_{p}}}}$.
4. $\underset{h\to 0}{\mathop{\lim }}\,||\Delta_{h}^{N}{{\left( f \right)}||_{\Phi ,~{{S}_{p}}}}=0$.

Proof

1. Let $\lambda >0$, then by (7), (9) and (11), we have

$||\Delta_{h}^{N}{{\left( f \right)}||_{\Phi ,~{{S}_{p}}}}\\=\underset{\lambda >0}{\mathop{\inf }}\,\frac{1}{\lambda }~{{S}_{p}}\left( _{h}^{N}\left( f \right) \right)\\=\underset{\lambda >0}{\mathop{\inf }}\,\frac{1}{\lambda }~{{S}_{p}}\left( \underset{i=1}{\overset{N}{\mathop \sum }}\,{{\left( -1 \right)}^{i}}\left( \begin{matrix}

   N  \\

   i  \\

\end{matrix} \right)f\left( x-ih \right) \right)\\ \le C\underset{\lambda >0}{\mathop{\inf }}\,\frac{1}{\lambda }~{{S}_{p}}\left( \underset{i=1}{\overset{\infty }{\mathop \sum }}\,{{\left( -1 \right)}^{i}}\left( \begin{matrix}

   N  \\

   i  \\

\end{matrix} \right)f\left( x-ih \right) \right) \\ \le C\underset{\lambda >0}{\mathop{\inf }}\,\frac{1}{\lambda }~\underset{i=0}{\overset{\infty }{\mathop \sum }}\,\left( \begin{matrix}

   N  \\

   i  \\

\end{matrix} \right){{S}_{p}}\left( {{\left( -1 \right)}^{i}}f\left( x-ih \right) \right) \\ \le C\underset{i=0}{\overset{\infty }{\mathop \sum }}\,\left( \begin{matrix}

   N  \\

   i  \\

\end{matrix} \right)\underset{\lambda >0}{\mathop{\inf }}\,\frac{1}{\lambda }~{{S}_{p}}\left( f\left( x-ih \right) \right)  \\ \le Ck\left( N \right){{||f}||_{\Phi ,~{{S}_{p}}}}$

2. By (7), (9) and (11), we have

$\Delta\left(\Delta _{h}^{N}\left( \Delta_{h}^{M}\left( f \right) \right) \right)\left( x \right) \\ =\underset{i=1}{\overset{N}{\mathop \sum }}\,{{\left( -1 \right)}^{i}}\left( \begin{matrix}

   N  \\

   i  \\

\end{matrix} \right)\left( _{h}^{M}\left( f\left( x-ih \right) \right) \right) \\ =\underset{i=1}{\overset{N}{\mathop \sum }}\,{{\left( -1 \right)}^{i}}\left( \begin{matrix}

   N  \\

   i  \\

\end{matrix} \right)\left( \mathop{\sum }_{i=1}^{M}{{\left( -1 \right)}^{i}}\left( \begin{matrix}

   M  \\

   i  \\

\end{matrix} \right)\left( f\left( x-ih \right) \right) \right) \\ =\underset{i=1}{\overset{N}{\mathop \sum }}\,\underset{i-1}{\overset{M}{\mathop \sum }}\,{{\left( -1 \right)}^{i}}\left( \begin{matrix}

   N  \\

   i  \\

\end{matrix} \right)\left( \begin{matrix}

   M  \\

   i  \\

\end{matrix} \right)f\left( x-hi \right) \\ =\underset{i=1}{\overset{N+M}{\mathop \sum }}\,{{\left( -1 \right)}^{i}}\left( \begin{matrix}

   N+M  \\

   i  \\

\end{matrix} \right)f\left( x-hi \right) \\ =\Delta_{h}^{N+M}\left( f\left( x \right) \right).$

3. By the above Lemma 1, we get

$||\Delta_{h}^{N+M}{{\left( f \right)}||_{\Phi ,\text{ }\!\!~\!\!\text{ }{{S}_{p}}}} \\ =||\Delta_{h}^{M}{{\left( \Delta_{h}^{N}\left( f \right) \right)}||_{\Phi ,\text{ }\!\!~\!\!\text{ }{{S}_{p}}}} \\ \le k\left( M \right)||\Delta_{h}^{N}{{\left( f \right)}||_{\Phi ,\text{ }\!\!~\!\!\text{ }{{S}_{p}}}} \\ \le {{2}^{\left( M \right)}}||\Delta_{h}^{N}{{\left( f \right)}_||{\Phi ,\text{ }\!\!~\!\!\text{ }{{S}_{p}}}}.$

4. Let $\epsilon >0$, choose $\delta =\delta \left( \epsilon ,N \right)$, by (11), we have $||\Delta_{h}^{N}{{\left( f_{J}^{w,b} \right)}||_{\Phi ,\text{ }\!\!~\!\!\text{ }{{S}_{p}}}}<\frac{\epsilon }{2}$, and

$||\Delta_{h}^{N}{{\left( f \right)}||_{\Phi ,~{{S}_{p}}}} \\ \le C\left[ ||\Delta_{h}^{N}{{\left( f-f_{J}^{w,b} \right)}||_{\Phi ,~{{S}_{p}}}}+||\Delta_{h}^{N}{{\left( f_{J}^{w,b} \right)}||_{\Phi ,~{{S}_{p}}}} \right] \\ \le C\left[ {{2}^{N}}||f-f{{_{J}^{w,b}}||_{\Phi ,~{{S}_{p}}}}+||\Delta_{h}^{N}{{\left( f_{J}^{w,b} \right)}||_{\Phi ,~{{S}_{p}}}} \right] \\ \le C\left[ {{2}^{N}}.\frac{\epsilon }{{{2}^{N+1}}}+\frac{\epsilon }{2} \right]=\epsilon $

In the following lemma, we give more generalized properties for our Orlicz modulus of smoothness.

Lemma 2

1. ${{\omega }_{N}}{{\left( f,\delta  \right)}_{\Phi ,~{{S}_{p}}}}$ is a positive nondecreasing continuous function of $\delta $ on $\left( 0,\infty  \right)$ and $\underset{\delta \to 0}{\mathop{\lim }}\,{{\omega }_{N}}{{\left( f,\delta  \right)}_{\Phi ,~{{s}_{p}}}}=0.$
2. ${{\omega }_{N}}{{\left( f+g,\delta  \right)}_{\Phi ,~{{S}_{p}}}}\le c\left( {{\omega }_{N}}{{\left( f,\delta  \right)}_{\Phi ,~{{S}_{p}}}}+{{\omega }_{N}}{{\left( g,\delta  \right)}_{\Phi ,~{{S}_{p}}}} \right)$
3. ${{\omega }_{N}}{{\left( f,\delta  \right)}_{\Phi ,~{{S}_{p}}}}\le {{2}^{N-M}}{{\omega }_{M}}{{\left( f,\delta  \right)}_{\Phi ,~{{S}_{p}}}}$
4. ${{\omega }_{N}}{{\left( f,\delta  \right)}_{\Phi ,~{{S}_{p}}}}~\le {{2}^{\left\{ N \right\}}}~{{||f}||_{\Phi ,~{{S}_{p}}}}$
5. ${{\omega }_{N}}{{\left( f,\delta  \right)}_{\Phi ,~{{S}_{p}}}}\le ~{{\omega }_{N}}{{\left( f,\overset{\acute{\ }}{\mathop{\delta }}\, \right)}_{\Phi ,~{{S}_{p}}}}$, for $\delta \le \overset{\acute{\ }}{\mathop{\delta }}\,$
6. ${{\omega }_{N}}{{\left( f,\gamma \delta  \right)}_{\Phi ,~{{S}_{p}}}}\le {{\left( 1+\gamma  \right)}^{k}}{{\omega }_{N}}{{\left( f,\delta  \right)}_{\Phi ,~{{S}_{p}}}},$ for$~0<\gamma \le 1.$

Proof

1. It is clear from Lemma 1
2. From (13), (12) and Lemma 1, we have

${{\omega }_{N}}{{\left( f+g,\delta  \right)}_{\Phi ,~{{S}_{p}}}} \\ =\underset{\left| h \right|\le \delta }{\mathop{\sup }}\,||\Delta_{h}^{N}{{\left( f+g \right)}||_{\Phi ,~{{S}_{p}}}} \\ =\underset{\left| h \right|\le \delta }{\mathop{\sup }}\,||\Delta_{h}^{N}\left( f \right)+\Delta_{h}^{N}{{\left( g \right)}||_{\Phi ,~{{S}_{p}}}} \\ \le c\left( \underset{\left| h \right|\le \delta }{\mathop{\sup }}\,||\Delta_{h}^{N}{{\left( f \right)}||_{\Phi ,~{{S}_{p}}}}+\underset{\left| h \right|\le \delta }{\mathop{\sup }}\,||\Delta_{h}^{N}{{\left( g \right)}||_{\Phi ,~{{S}_{p}}}} \right) \\ \le ~c\left( {{\omega }_{N}}{{\left( f,\delta  \right)}_{\Phi ,~{{S}_{p}}}}+{{\omega }_{N}}{{\left( g,\delta  \right)}_{\Phi ,~{{S}_{p}}}} \right)$

3. Again by (11), (12) and Lemma 1

${{\omega }_{N}}{{\left( f,\delta  \right)}_{\Phi ,~{{S}_{p}}}} \\ =\underset{\left| h \right|\le \delta }{\mathop{\sup }}\,||\Delta_{h}^{N}{{\left( f \right)}||_{\Phi ,~{{S}_{p}}}} \\ =\underset{\left| h \right|\le \delta }{\mathop{\sup }}\,||\Delta_{h}^{N-M}{{\left( _{h}^{M}\left( f \right) \right)}||_{\Phi ,~{{S}_{p}}}} \\ \le k\left( N-M \right)\underset{\left| h \right|\le \delta }{\mathop{\text{sup}}}\ ||\Delta_{h}^{M}{{\left( f \right)}||_{\Phi ,~{{S}_{p}}}} \\ \le {{2}^{N-M}}{{\omega }_{M}}{{\left( f,\delta  \right)}_{\Phi ,~{{S}_{p}}}}.$

4. By Lemma 1

${{\omega }_{N}}\left( f,\delta  \right) \\ =\underset{\left| h \right|\le \delta }{\mathop{\text{sup}}}\ ||\Delta_{h}^{N}{{\left( f \right)}||_{\Phi ,~{{S}_{p}}}}\le {{2}^{N}}\underset{\left| h \right|\le \delta }{\mathop{\text{sup}}}\,{{||f||}_{\Phi ,~{{S}_{p}}}} \\ \le {{2}^{N}}~{{||f||}_{\Phi ,~{{S}_{p}}}}.$

5. Let $\delta <\overset{\acute{\ }}{\mathop{\delta }}\,$, then by (12), it is clear that

${{\omega }_{N}}{{\left( f,\delta  \right)}_{\Phi ,~{{S}_{p}}}} \\ =\underset{\left| h \right|\le \delta }{\mathop{~\sup }}\,||\Delta_{h}^{N}{{\left( f \right)}||_{\Phi ,~{{S}_{p}}}} \\ \le \underset{\left| h \right|\le \overset{\acute{\ }}{\mathop{\delta }}\,}{\mathop{~\sup }}\,||\Delta_{h}^{N}{{\left( f \right)}||_{\Phi ,~{{S}_{p}}}}={{\omega }_{N}}{{\left( f,\overset{\acute{\ }}{\mathop{\delta }}\, \right)}_{\Phi ,~{{S}_{p}}}}$

6. Noting that $\gamma \delta \le \delta $, then

${{\omega }_{N}}{{\left( f,\gamma \delta  \right)}_{\Phi ,~{{S}_{p}}}} \\ \le {{\omega }_{N}}{{\left( f,\delta  \right)}_{\Phi ,~{{S}_{p}}}} \\ \le {{\left( 1+\gamma  \right)}^{k}}{{\omega }_{N}}{{\left( f,\delta  \right)}_{\Phi ,~{{S}_{p}}}}.$

For any input $x=\left( {{x}_{i}} \right)_{i=1}^{d}$, define the oprater ${{\mathcal{L}}_{x}}:{{L}_{\Phi ,{{S}_{p}}}}\left( {{I}^{d}} \right){{\mathbb{R}}^{+}}$ as follow

${{\mathcal{L}}_{t}}\left( g\left( u \right) \right)=\mathop{\sum }_{i=1}^{d}g\left( {{x}_{i}} \right){{\delta }_{i}}\left( u \right)$      (17)

where, $g\in {{L}_{\Phi ,{{S}_{p}}}},u\in I$ and ${{\delta }_{i}}~$is given by

    ${{\delta }_{i,g}}\left( u \right)=\mathop{\sum }_{i=2}^{d}{{\left( -1 \right)}^{N-i}}\left( \begin{matrix}

   N  \\

   i  \\

\end{matrix} \right)g\left( {{t}_{i}} \right)\sigma \left( {{t}_{i}}-u \right)$         (18)

where, $\sigma :~I\to I~$is the ReLU activation function defined by (16).

In fact, Eq. (17) is a convolutional operator between the function g and the activation function $\sigma $. Moreover, $x=\left( {{x}_{i}} \right)_{i=1}^{d}~$maybe not only the input, but also any neuron acts as input in the network. This amount of convolutions implies high dimensions of the hidden layer when the outputs of neuron clusters are combined. The purpose of the pooling layers is to reduce the big widths that may cause overfitting. Another procedure that can reduce the dimensions, called downsampling, is discussed in the following section.

The following lemma arises the role of convolutions [44] concluded that ${{W}^{\left( k \right)}}$ equals ${{\Im }^{\left( {{J}_{k}} \right)}}~\cdots ~{{\Im }^{\left( {{J}_{k-1}}+1 \right)}}$.

Lemma 3

${{\Im }^{\left( {{\text{J}}_{\text{k}}},{{\text{J}}_{\text{k}-1}}+1 \right)}}={{\Im }^{\left( {{\text{J}}_{\text{k}}} \right)\text{ }\!\!~\!\!\text{ }}}\text{ }\!\!~\!\!\text{ }\cdots \text{ }\!\!~\!\!\text{ }\Im {{\text{ }\!\!~\!\!\text{ }}^{\left( {{\text{J}}_{\text{k}-1}}+2 \right)}}\Im {{\text{ }\!\!~\!\!\text{ }}^{\left( {{\text{J}}_{\text{k}-1}}+1 \right)}}$ where $k=1,\cdots ,\ell $.

Construction of filter masks is studied by Zhou [48] in his following lemma, given convolution$~{w^{\left( \text{J} \right)}}\text{*}\cdots \text{*}{w^{\left( 1 \right)}}$ resulted from factorizing $W$ as follows

Lemma 4

For $s\ge 2$, $M\ge 0$, any sequence $W=\left( {w_{k}} \right)_{-\infty }^{\infty }$ in $\left\{ 0,~\cdots ,~M \right\}$, there is a finite sequence $\left\{ {w^{\left( j \right)}} \right\}_{j=1}^{J}$ with nonzero elements in$~\left\{ 0,~\cdots ,~s \right\}$, where $J<\frac{M}{s-1}+1$ that satisfies

$W={w^{\left( j \right)}}*~\cdots *{w^{2}}*{w^{1}}$                (25)

To fit the data with the space under study, bias vectors are selected in appropriate format in accordance to the convolutional matrices ${{\Im }^w}$, let $h:~\Omega \rightarrow {{R}^{D}}$, set

${{||h}||_{\Phi ,~{{S}_{p}}}}=\underset{\lambda >0}{\mathop{\text{inf}}}\,~\frac{1}{\lambda }{{\left( I_{\Phi }^{2}\left( \lambda h \right)+1 \right)}^{\frac{1}{p}}}$                     (26)

Also, we need to denote ${{||w||}_{\Phi ,~{{S}_{p}}}}=\underset{\lambda >0}{\mathop{\inf }}\,~\frac{1}{\lambda }{{\left( I_{\Phi }^{2}\left( \lambda {w_{k}} \right)+1 \right)}^{\frac{1}{p}}}$.

Then we immediately see how convolutional matrices are defined ${{\Im }^{\left( j \right)}}={{\Im }^{{w^{\left( j \right)}}}}$ that for any $h~:\Omega \rightarrow {{\mathbb{R}}^{{{d}_{j}}-1}}$, we denoted

${{||\Im }^{\left( j \right)}}{{h}||_{\Phi ,~{{S}_{p}}}}\le {{||w||}_{\Phi ,~{{S}_{p}}}}{{||h||}_{\Phi ,~{{S}_{p}}}}$                  (27)

From our previous study AL-Janabi and Almurieb [13], we use some thoughts from DCNNs without downsampling and select biases small enough for the vectors ${{\Im }^{\left( j \right)}}{{h}^{\left( j-1 \right)}}\left( x \right)-{{b}^{\left( j \right)}}$ to get non-negative entries.

Lemma 5

For any $k\in \left\{ 1,\cdots ,\ell  \right\}$, there exists $B\in {{R}^{+}}$, $\hat{B}\in \left[ -B,B \right]$, then ${{||h}^{{{j}_{k-1}}}}-\hat{B}{{1}_{{{d}_{{{j}_{k-1}}}}}}||_{\Phi ,~{{S}_{p}}}\le B$.

Proof

Set No space

$b{{~}^{\left( {{J}_{k-1}}+1 \right)}}=\hat{B}\Im {{\text{ }\!\!~\!\!\text{ }}^{\left( {{J}_{k-1}}+1 \right)}}{{1}_{{{d}_{{{J}_{k-1}}}}}}-B{{||w}^{\left( ({{J}_{k-1}}+1 \right)}||}_{\Phi ,~{{s}_{p}}}{{1}_{{{d}_{{{J}_{k-1~}}}}+1}}$.

To prove $b_{{{s}^{\left( j \right)}}+1}^{\left( j \right)}=b_{{{s}^{\left( j \right)}}~+2}^{\left( j \right)}=\cdots =b_{{{d}_{j-1}}}^{\left( j \right)}$

From

${{b}^{\left( j \right)}}= B \left( \Pi _{p={{J}_{k-1}}+1}^{j-1}{{||w}^{\left( p \right)}}||_{\Phi ,~{{S}_{p}}} \right){{\Im }^{\left( j \right)}}{{1}_{{{d}_{j-1}}}} \ - B \left( \Pi _{p={{J}_{k-1}}+1}^{j-1}{{||w}^{\left( p \right)}}||_{\Phi ,~{{S}_{p}}} \right){{1}_{{{d}_{j-1+{{s}^{\left( j \right)}}}}}}$

For $j={{J}_{k-1}}+2,\cdots ,{{J}_{k-1}}$, then for ${{J}_{k-1}}<j<{{J}_{k}}$ and $i={{s}^{\left( j \right)}}+1,\cdots ,{{d}_{j-1}}$, we have${{\left( {{\Im }^{\left( j \right)}}{{1}_{{{d}_{j-1}}}} \right)}_{i}}=\underset{p=1}{\overset{{{d}_{j-1}}}{\mathop \sum }}\,{{\left( {{\Im }^{\left( j \right)}} \right)}_{i,p}}=\underset{p=1}{\overset{{{d}_{j-1}}}{\mathop \sum }}\,{{\left( {{w}^{\left( j \right)}} \right)}_{i-p}}$.

Notice that w^(j) in $\left\{ 0,\cdots ~,~{{s}^{\left( j \right)}} \right\}$. $p\in \left( -\infty ,0\left] \cup  \right[{{d}_{j-1}}+1,\infty  \right)$ we have $i-p\in \left[ {{s}^{\left( j \right)}}+1,\infty  \right)$ which implies that ${{w}^{\left( j \right)}}_{i-p}=0$. So

${{\left( {{\Im }^{\left( j \right)}}{{1}_{{{d}_{j-1}}}} \right)}_{i}}=\underset{p=-\infty }{\overset{\infty }{\mathop \sum }}\,{{w}^{\left( j \right)}}_{i-p}=\underset{p=-\infty }{\overset{\infty }{\mathop \sum }}\,{{w}_{p}}^{\left( j \right)}\forall i={{s}^{\left( j \right)}}+1,\cdots ,{{d}_{j-1}}$

Now we prove

  ${{h}^{\left( j \right)}}\left( x \right)={{\Im }^{\left( j \right)}}\ldots {{\Im }^{\left( {{J}_{k-1}}+1 \right)}}\left( {{h}^{{{J}_{k-1}}}}\left( x \right)-\hat{B}{{1}_{{{d}_{{{J}_{k-1}}}}}} \right)+ \ B \left( \underset{p={{J}_{k-1}}+1}{\overset{j}{\mathop \prod }}\,{{||w}^{\left( p \right)}}||_{\Phi ,~{{s}_{p}}} \right)~{{1}_{{{d}_{j}}}}$

By induction, for $j={{J}_{k-1}}+1$, we have​

${{\Im }^{\left( {{J}_{k-1}}+1 \right)}}{{h}^{\left( {{J}_{k-1}} \right)}}\left( x \right)-{{b}^{\left( {{J}_{k-1}}+1 \right)}}={{\Im }^{\left( {{J}_{k-1}}+1 \right)}}\left( {{h}^{{{J}_{k-1}}}}\left( x \right)-\hat{B}{{1}_{{{d}_{{{J}_{k-1}}}}}} \right)+B{{||w}^{\left( {{J}_{k-1}}+1 \right)}}||_{\Phi ,~{{s}_{p}}}{{1}_{{{d}_{{{J}_{k-1~}}}}+1}}$

the use Sigmoid $\sigma $ is identical to the identity function on $\left[ 0,~\infty  \right)$, hence

${{h}^{\left( {{J}_{k-1}}+1 \right)}}\left( x \right)={{\Im }^{\left( {{J}_{k-1}}+1 \right)}}\left( {{h}^{{{J}_{k-1}}}}\left( x \right)-\hat{B}{{1}_{{{d}_{{{J}_{k-1}}}}}} \right)+B{{||w}^{\left( {{J}_{k-1}}+1 \right)}}||_{\Phi ,~{{s}_{p}}}{{1}_{{{d}_{{{J}_{k-1~}}}}+1}}$

for $j=1$ verifies.

Suppose that it is holds for ${{J}_{k-1}}+1\le j\le {{J}_{k-1}}$, then ${{d}_{j-1}}+{{s}^{\left( j \right)}}={{d}_{j}}$.

Through the induction hypothesis and selection (*) of the bias vector, we have

${{h}^{\left( j \right)}}\left( x \right)=\sigma \left( {{\Im }^{\left( j \right)}}{{\Im }^{\left( j-1 \right)}}\cdots {{\Im }^{\left( ~{{J}_{k+1}} \right)}}\left( {{h}^{\left( ~{{J}_{k-1}} \right)}}\left( x \right)-\hat{B}{{1}_{{{d}_{{{J}_{k-1}}}}}} \right)+B\left( \underset{p={{J}_{k-1}}+1}{\overset{j}{\mathop \prod }}\,{{||w}^{\left( p \right)}}||_{\Phi ,~{{s}_{p}}} \right){{1}_{{{d}_{j}}}} \right)$

Now to prove

$h^{\left(J_k\right)}(x)=M^{(k)} \Im^{\left(J_k J_{k-1}+1\right)}\left(h^{\left(J_{k-1}\right)}(x)-\widehat{B} \mathbf{1}_{d_{J_{k-1}}}\right) +B\left( \underset{p={{J}_{k-1}}+1}{\overset{{{J}_{k}}}{\mathop \prod }}\,{{||w}^{\left( p \right)}}||_{\Phi ,~{{s}_{p}}} \right){{1}_{{{d}_{{{J}_{k}}}}}}$

When

${{b}^{\left( j \right)}}=B\left( \Pi _{p={{J}_{k-1}}+1}^{j-1}{{||w}^{\left( p \right)}}||_{\Phi ,~{{s}_{p}}} \right){{\Im }^{\left( j \right)}}{{1}_{{{d}_{j-1}}}}-B$,

Then

${{\Im }^{\left( {{J}_{k}} \right)}}{{h}^{\left( ~{{J}_{k-1}} \right)}}\left( x \right)-{{b}^{\left( {{J}_{k}} \right)}}={{\Im }^{\left( {{J}_{k}} \right)}}\cdots {{\Im }^{\left( ~{{J}_{k-1}} \right)}}\left( {{h}^{\left( ~{{J}_{k-1}} \right)}}\left( x \right)-\hat{B}{{1}_{{{d}_{{{J}_{k-1}}}}}} \right) + B\left( \underset{p={{J}_{k-1}}+1}{\overset{{{J}_{k}}}{\mathop \prod }}\,{{||w}^{\left( p \right)}}||_{\Phi ,~{{s}_{p}}} \right)~{{1}_{{{d}_{{{J}_{k}}}}}}$