Hybrid Transceiver Schemes for Spatial Multiplexing and Diversity in MIMO Systems

— In this article, we present hybrid multiple-input multiple-output (MIMO) transceiver schemes (HMTS) that combine transmit diversity and spatial multiplexing, thus achieving at the same time the two possible spatial gains offered by MIMO systems. For these transceivers, a modiﬁcation in the interference nulling-and-cancelling algorithm used in traditional MIMO schemes is proposed. We propose a novel MIMO receiver architecture to cope with the hybrid transmission schemes by jointly performing the tasks of interference cancellation and space-time decoding. Both successive and ordered successive detection strategies are considered in the formulation of the receivers. Our simulation results show satisfactory performance of the HMTS when combined with the proposed receivers, outperforming the standard vertical Bell laboratories layered space-time system in terms of bit/symbol error rate, while providing higher spectral efﬁciencies than a pure space-time block code system.

block codes (STBC) [4,5], are concerned with diversity gain.In other words, their objective is to increase the link reliability against fading.Apart from these two gains, it is also possible to achieve coding gain, as in the case of space-time trellis codes (STTC).This topic will be left to a future investigation.In this article, we focus on the spatial multiplexing and diversity gains.
More specifically, if there are M transmit and N receive antennas, generically denoted as (M Tx-N Rx), with sufficient signal scattering and antenna spacing, there are M N independent links between the transmitter and the receiver.In this situation it is possible to provide an M N -fold protection against channel fading.This protection is called diversity gain and the number of independent links is the diversity order.On the other hand, there are min(M, N ) degrees of freedom, which can be used to spatially multiplex data for increase spectral efficiency.This gain in multiplexing symbols through the MIMO wireless channel is known as spatial multiplexing gain.MIMO structures designed specifically to maximize the spatial multiplexing gain can provide high spectral efficiency that increases almost linearly with the number of degrees of freedom [1][2][3].
Most MIMO schemes are designed to achieve just one of two available gains from these systems, i.e., either spatial multiplexing gain or spatial diversity gain (omitting the coding gain).There is, however, a trade-off: a compromise between spectral efficiency and diversity gain can be expected when considering different MIMO implementations.Recent works [6,7], deal with the trade-off present in the MIMO system using an information-theoretic approach.However, none of them suggested practical structures capable of achieving an optimal trade-off between spatial multiplexing and diversity gains.
One solution in this direction was proposed with a modification in the VBLAST scheme, called diagonal BLAST [1], in which the transmitted symbols are multiplexed in all the transmit antennas available, but in different time instants.Unfortunately, this solution brings a considerable delay in order to achieve a diversity gain, and thus is not very practical.
Hybrid MIMO transmission schemes (HMTS) arise as a solution to jointly achieve spatial multiplexing and diversity gains.With HMTS, it is possible to considerably increase the data rate while keeping a satisfactory link quality in terms of bit error rate (BER) or symbol error rate (SER).In fact, HMTS apply pure diversity schemes (e.g.STBC) jointly with pure spatial multiplexing schemes (e.g.VBLAST), so that parts of the data are space-time coded across some antennas, and these parts are combined in layers, using a VBLAST approach.As spatially-multiplexed layers see each other as interference, interference cancellation algorithms similar to that employed in VBLAST is mandatory in the receiver.
In this article, we present HMTS designed for three and four transmit antennas, motivated by current practical limitations in the feasible number of antennas for MIMO systems.However, the concept can be extended to higher dimensions using the same reasoning presented in this work.We compute the spectral efficiency of the proposed HMTS, which supports the claim that they have better spectral efficiency than schemes that focus solely on the diversity gain.We also present several simulation results illustrating the satisfactory performance of the proposed schemes as compared to conventional ones, in terms of both link reliability and spectral efficiency.
This article is organized as follow.In section II, we present the MIMO system and channel model considered.In section III we review conventional MIMO transceiver schemes, while in section IV we propose the HMTS.Section V presents performance results.In section VI we state some conclusions and possible future work.

II. MIMO SYSTEM AND CHANNEL MODELS
In this article, we consider a transmitter equipped with an M -element antenna array and a receiver equipped with an N -element antenna array, as seen in Fig. 1.The transmitted signals are assumed to go through a random channel matrix H.The wireless channel is assumed to have rich-scattering and flat-fading.The fading between each transmit and receive antenna pair is assumed to be independent and the entries of H are circularly symmetric Gaussian random variables.The quasi-static block fading model is assumed; in other words, the channel matrix H is randomly generated, but remains constant during the transmission of one space-time code word of length K.A new random channel matrix, independent of the previous one, is then generated for each new space-time code word.We disregard the frequency selectivity of the channel since it is well-known that a frequency selective channel can be converted into parallel flat-fading channels by inverse fast Fourier transform (IFFT) and fast Fourier transform (FFT).Thus, all methods presented here are also applicable to frequency selective channels.Furthermore, we assume i.i.d circularly symmetric Gaussian noise samples.For all the MIMO transmission schemes, we assume that the total transmit power is fixed (normalized to 1) and equally divided across the transmit antennas.Ideal symbol timing is assumed at the receiver.Thus, we can relate the transmit and receive symbols through the relation at time k in complex baseband form and at the symbol rate where x ∈ C N denotes the vector of complex received symbols during any given channel use, s ∈ C M denotes the vector of the complex transmitted symbols, H ∈ C N ×M denotes the channel matrix, v ∈ C N is the zero-mean, unit variance and complex-Gaussian distributed noise that is spatially and temporally white, and ρ is the signal-to-noise ratio (SNR).The entries of channel matrix H and the transmitted vector s are assumed to have unit variance, implying that and where tr(•) denotes the trace of the matrix and E[•] is the expectation operator.The normalization factor ρ M in (1) guarantees that the SNR at each receiver antenna is independent of M .

III. CONVENTIONAL MIMO TRANSCEIVER SCHEMES
In general, MIMO architectures can be classified in one of three groups depending on the provided gains: pure diversity schemes, pure multiplexing schemes and hybrid MIMO schemes.Heretofore, we denote the pure diversity schemes and pure multiplexing schemes as conventional MIMO transceiver schemes.As their names imply, conventional MIMO transmission structures provide either diversity gain or spatial multiplexing gain, but not both.In this section, we briefly describe the conventional MIMO transceivers schemes.

A. Pure diversity schemes
Space-time codes (STC) [5] are a well-known technique that provides diversity gain.Space-time codes use channel coding techniques combined with multiple transmit antennas, introducing temporal and spatial correlations into signals transmitted from different antennas, thus increasing the diversity order at the receiver.Two techniques widely used for STC are space-time block codes (STBC) and space-time trellis codes (STTC).In the latter, when the number of transmit antennas is fixed, the decoding complexity (measured by the number of trellis states at the decoder) increases exponentially as a function of the number of antennas of the code.In addressing the issue of decoding complexity, Alamouti [4] discovered a remarkable STBC scheme, denoted here as G2, for transmission with two antennas in quasi-static and flat-fading channels.Due to its very simple encoding and decoding, Alamouti's scheme is being considered for the universal mobile telecommunications system (UMTS) standards [8].
The success of G2 spurred a search for new schemes of different rates and for more transmit antennas.We now describe some of these STBC schemes [4,5], which will be considered in this work.We will follow the notation presented in [5] in which Tarokh named his schemes for M > 2 as: • the letter G represents schemes achieving the data symbols rate of 1/2 per channel use (pcu); • the letter H represents schemes achieving the data symbols rate of 3/4 pcu; • Following the letters (G or H) is the number of transmit antennas of the schemes.For example, H3 is a scheme with rate 3/4 designed for 3 transmit antennas, while G4 is a scheme with rate 1/2 designed for 4 transmit antennas.Following the literature, we will denote Alamouti's STBC by G2, even though it achieves 1 symbol pcu.
1) G2 STBC scheme: in this scheme, two data symbols s 1 and s 2 are simultaneously transmitted by different antennas at a given symbol period k, where s 1 is transmitted by antenna one and s 2 is transmitted by antenna two.In the next symbol period k + 1, antenna one transmits −s * 2 and antenna two transmits s * 1 .The transmitted signals can be organized in the equivalent space-time coding matrix where the rows of S G2 [k, k + 1] correspond to the symbol periods, its columns correspond to the transmit antennas and (•) * means complex conjugation.Due to the orthogonality of the transmit matrix S G2 [k, k + 1], the maximum-likelihood (ML) detection involves a simple linear operation in the receiver and can be used to detect the transmit symbols s 1 and s 2 , assuming that the channel is static during two consecutive symbol periods, k and k + 1.The orthogonality characteristic of S is based on the orthogonal designs [5] such that for orthogonal STBC G2 we have where (•) H is the Hermitian matrix and I M is an M × M identity matrix.This characteristic makes possible the ML detection of the transmitted symbols regarding just one receiver antenna, since the decision statistics for the desired signal s m , m = 1, . . ., M , is independent of the other transmitted signals.Therefore, all STBC schemes can be decoded with just one receiver antenna with a low complexity based on linear processing.
Since the G2 scheme multiplexes K = 2 information symbols (s 1 and s 2 ) in T = 2 consecutive channel uses, the data symbol rate of this scheme is equal to K/T = 1 symbol pcu, while the effective spectral efficiency of this scheme is equal to η = (K/T ) • log 2 M = log 2 M bps/Hz, where M is the cardinality of the modulation scheme considered.Schemes that achieve K/T = 1 are also known as full-rate (FR) schemes.
2) G3 STBC scheme: in this scheme the transmitted signals can be organized in the equivalent space-time coding matrix As with G2, a simple linear operation in the receiver can be used to detect the transmit symbols s 1 , s 2 , s 3 and s 4 .However, in this case, the channel needs to be static during eight consecutive symbol periods, k, . . ., k + 7. Since the G3 scheme multiplexes K = 4 information symbols (s 1 , s 2 , s 3 and s 4 ) in T = 8 consecutive channel realizations, the effective spectral efficiency of this scheme is equal to η = (1/2) • log 2 M bps/Hz.
3) G4 STBC scheme: this scheme also has rate 1/2, but it is designed for four transmit antennas and its transmitted signals can be organized in the equivalent space-time coding matrix The effective spectral efficiency of this scheme is equal to η = (1/2) • log 2 M bps/Hz.4) H3 STBC scheme: finally, the STBC H3 has an equivalent space-time coding matrix given by ) Since the H3 scheme multiplexes K = 3 information symbols (s 1 , s 2 and s 3 ) in T = 4 consecutive channel realizations, the effective spectral efficiency of this scheme is equal to η = (3/4) • log 2 M bps/Hz.

B. Pure multiplexing scheme
Another approach for multiple-antenna transmission is to focus on the maximization of the spectral efficiency.
Well-known schemes proposed with this focus are the Bell laboratories layered space-time (BLAST) schemes, such as the vertical-BLAST (VBLAST) and diagonal-BLAST [2].In the VBLAST scheme, all the antennas are used to multiplex different symbols in each symbol period.In this scheme each different multiplexed symbol is defined as a layer.For instance, in the case of three transmit antennas we have three layers.The transmitted signals at time instant k, considering three transmit antennas, can be organized in the equivalent space-time coding matrix As spatially-multiplexed symbols cause interference in each other, signal processing is mandatory at the receiver in order to cancel interference.In the following we describe both linear and non-linear approaches for interference cancellation.
1) Linear detector: the output of a linear detector (LD) considering (1) applied to the received signal can be written as Since the desired output is s[k], we define the error vector at the output of the spatial filter as Different values of W lead to different detectors.In this article, we consider the minimum mean-square error (MMSE) filter, which minimizes the error variance, written as The linear detector W that minimizes the cost function J MMSE is given by [9] (omitting the index [k] without loss of generality) where R xx = E{xx H } and R sx = E{sx H } are the input covariance matrix and a cross-correlation between the desired output s and the received vector x at time k, respectively.
2) Non-linear detector: the operation of mitigating the interference with linear signal processing is normally referred to as nulling.However, a superior performance can be reached when a non-linear spatial-processing approach is used.A common non-linear detector is based on interference cancellation (IC), wherein the contribution of the detected symbols to the received signal is reconstructed and subtracted.Assuming correct decisions, the resulting signal is free from the interference of the detected symbols, yielding better estimates of the remaining symbols.
One particularly successful IC algorithm is called successive interference cancellation (SIC).In SIC, the layers are detected sequentially.Initially, the received signal x[k] goes through a linear detector for layer 1, whose output is used to produce a hard estimate of the symbols at this layer, ŝ1 [k].Then, the contribution of layer 1 to the received signal is estimated and cancelled, generating the signal x 2 [k].The process is then repeated.In general, at the i-th layer, the signal x i [k], hopefully free from the interference of layers j < i, goes through a linear detector that tries to mitigate the interference from layers j > i.A hard estimate of the symbol at this layer, ŝi [k], is then produced, based on the output of this linear detector.Then, the contribution of this layer to the "received signal" x i [k] is estimated and cancelled.This procedure yields a modified received signal given by where h i is the i-th column of the matrix channel H corresponding to the channel gains associated to layer i, and ŝi [k]h i [k] represents the estimated interference from the i-th layer.The result is that x i+1 [k] is free from the interference coming from layers 1, . . ., i.This signal is then fed into the linear detector for the (i + 1)-th layer.This technique is also known as nulling and cancelling algorithm [10].
The performance of SIC can be improved if the layers are detected in an appropriate order, resulting in ordered successive interference cancellation (OSIC).Indeed, one of the disadvantages of SIC is that the signal associated with the first detection layer may exhibit a lower received SNR than that of the other layers.This may increase the probability of detection errors, which can propagate through the serial detection process, degrading performance of the overall receiver.This problem can be mitigated if the layers are ordered by decreasing SNR, so that the first layer to be detected is that with the higher SNR [10].

IV. HYBRID MIMO TRANSCEIVER SCHEMES (HMTS)
As mentioned in the introduction, the use of multiple transmit and receive antennas may result in great capacity gains.Indeed, in a rich scattering environment the deployment of antenna arrays at both link-ends results in a capacity that increases almost linearly with the minimum number of antennas [1][2][3].Such a capacity increase is known as spatial multiplexing gain.MIMO antenna systems may also provide diversity gain, which is a measure of robustness against fading [5].There is, however, a trade-off: the diversity gain can only be increased if the multiplexing gain is sacrificed like shown in [6].The conventional systems described in section III lie in extreme points in the trade-off curve [6]: they provide only multiplexing or diversity gains.In this section, we described hybrid MIMO transceiver schemes (HTMS), which lie in intermediate points in the tradeoff curve, providing both types of gains.
In general, the transmission process of a hybrid scheme can be divided in layers, somewhat like VBLAST.However, in contrast to VBLAST, in the hybrid case a layer may consist of the stream of symbols at the output of a STBC, which is sent to a group of antennas, or of an uncoded stream, which is transmitted from a single antenna.Based on this concept of layers, hybrid MIMO transceiver schemes combine pure diversity schemes (e.g.STBC) with pure spatial multiplexing schemes (e.g.VBLAST).In hybrid systems, some layers are space-time coded across two, three or four antennas.For the remaining layers, a VBLAST approach is used.With this idea, hybrid MIMO schemes achieve a compromise between spatial multiplexing and transmit diversity gains.The basic idea behind these structures is to combine array processing and space-time coding, as first presented in [11].In [11] Tarokh et al. combined STTC and array processing by partitioning antennas at the transmitter into small groups.The signal transmitted in each group of antennas goes through a given STTC, called by authors component codes.At the receiver, the signals from each STTC are separated by a non-linear processing technique that suppresses signals transmitted from other groups of antennas, by treating them as interference.Then, the STTC are individually decoded.Tarokh's et al. idea involves a fixed transmission structure in [11], the authors did not consider adapting the transmitter to the channel conditions.However, since the wireless channel is random, using a fixed structure in some cases could represent a waste of the resources.
The idea behind HMTS is similar to that in [11].However, in our case we consider a family of transmission structures for three and four transmit antennas that are capable of achieving at the same time spatial diversity and multiplexing gains.Thus, our approach is naturally an adaptable structure, changing the focus from diversity to multiplexing according to the MIMO channel characteristics and performance objectives.Another difference between our approach and that presented in [11] is that we consider STBC instead STTC.Thus, in general, our system has a lower decoding complexity.
In the remainder of this section we present some specific HMTS.The notation for a particular HMTS is based on the notation of the STBC used by the HMTS (e.g.G2 or G3), while each uncoded streams following the VBLAST scheme is denoted in the label of the HMTS as +1.For example, the system designed for three transmit antennas consisting of two layers, one space-time coded with the G2 scheme and another uncoded layer following the VBLAST scheme, is denoted G2+1.

A. HMTS designed for three transmit antennas: G2+1
We now present the first proposed hybrid MIMO strategy, called G2+1.This hybrid scheme, whose structure is shown in Fig. 2(a), employs a three-element transmit antenna array with two spatial multiplexing layers.A standard G2 (Alamouti's) space-time block code is used at the first layer; the other layer is not space-time-coded, following the VBLAST approach.In the G2+1 scheme, the transmitted signals can be organized in the equivalent space-time coding matrix where the spatial dimension varies column-wise and the temporal dimension row-wise.From (15), it can be seen that K = 4 information symbols (two from each multiplexing layer) are transmitted in T = 2 consecutive channel uses.Thus, the effective spectral efficiency of this scheme is equal to η = 2 • log 2 M bps/Hz.

B. HMTS designed for four transmit antennas
In this section we present three HMTS designed for four transmit antennas.
1) G2+G2: the second HMTS, called G2+G2, is shown in Fig. 2(b).It employs a four-element transmit antenna array with two vertical-layered G2 space-time coding schemes.Observe that the four transmit antennas are divided into two space-time coding groups of two antennas each.The transmitted signals can be organized in an equivalent space-time coding matrix given by From ( 16), it can be seen that K = 4 information symbols (two from each multiplexing layer) are transmitted in T = 2 consecutive channel realizations.Thus, the effective spectral efficiency of this scheme is equal to η = 2 • log 2 M bps/Hz.Compared to the standard G4 space-time coding, the G2+G2 scheme achieves twice the data symbols rate.
2) G3+1: Fig. 2(c) depicts the third HMTS considered in this work.The four transmit antennas are now divided into two multiplexing layers, where the first one consists of three antennas that are space-time coded using G3 code [5].The equivalent space-time coding matrix for this hybrid scheme is given by From ( 17), we observe that K = 12 information symbols (four from the first layer and eight from the second one) are transmitted in T = 8 consecutive channel uses.Thus, the effective spectral efficiency of this scheme is equal to η = 1.5•log 2 M bps/Hz.This represent three times the spectral efficiency of G4.
3) G2+1+1: the fourth HMTS scheme is called G2+1+1 and is depicted in Fig. 2(d).Again, four transmit antennas are employed.As can be seen from the figure, this scheme consists of three spatial multiplexing layers; the first layer is space-time coded using G2, and the remaining are transmitted using VBLAST.The equivalent space-time coding matrix for the G2+1+1 scheme is given by In this HMTS, K = 6 information symbols (two from the first layer and four from the uncoded ones) are transmitted in T = 2 consecutive channel uses.Thus, the effective spectral efficiency of this scheme is equal to η = 3 • log 2 M bps/Hz.Compared to conventional G4 space-time code, this hybrid scheme achieves three times its data rate.Furthermore, the G2+1+1 scheme offers a 50% increase in spectral efficiency compared to the G2+G2 scheme.

C. HMTS as linear dispersion codes
As shown in (15) to (18) the MIMO transmission structures can be organized through a space-time equivalent matrix.The space-time equivalent matrix S defines the transmitted symbols in each antenna per signaling interval.Thus, the matrix S has dimension T × M , where the columns represent the transmit antennas and the rows represent the signaling interval.
In [12], the authors presented codes that generalize all linear MIMO transmission structures, known as linear dispersion codes (LDC).Through LDC we can represent the equivalent matrix S of all linear MIMO space-time transmission structures (e.g.STBC, VBLAST-based structures and also the HMTS).The LDC approach is described in the sequel.
Let K be the number of transmitted symbols in the T signaling intervals, and let s k be one of these symbols, given by where α k is the real part, β k is the imaginary part of s k and j = √ −1.The transmission matrix of the LDC is given by where C k e D k are the matrices that characterize the linear MIMO transmission structure.Hereafter, without loss of generality we will omit the term [k, . . ., k As in the case of the conventional MIMO transmit schemes it is also possible to represent the HMTS as a LDC.We will describe here the G2+1 case, but the extension for other HMTS is straightforward.
For HMTS G2+1 we have the following parameters The matrices A k and B k are given by Replacing A k and B k in (23) we have Therefore, through an appropriate choice of the matrices A k and B k , it is possible to represent all linear MIMO transmission strategies, including the HMTS as shown here to the case of HMTS G2+1.

D. Modified interference cancellation algorithm for the hybrid MIMO transceiver schemes
By the definition of the HMTS, all the proposed HMTS have at least two layers, at least one of which is space-time block coded.Further, they all employ orthogonal STC, whose ML detection involves simple linear operations in the receiver.We now propose a receiver for the HMTS that combines SIC algorithm with the simplicity of ML detection of an orthogonal STC.In fact, we adapt the IC algorithm in such a way that the orthogonal structure of the space-time code is preserved as much as possible in its output signal.The general structure of the receiver is shown in Fig. 3.We will explain this structure for the G2+1 case.The extension to other hybrid schemes is straightforward.
In the case of G2+1, we have two layers: a standard G2 space-time block code at the first layer and a non-space-time-coded layer.Being more robust, the G2 layer is detected first.In this case, the error vector at the output of the MIMO-MMSE spatial filter shown in Fig. 3 is given by e where Here, s 1 is the G2-encoded signal, and H d corresponds to the first two columns of the channel matrix H. Contrarily to the classical MIMO-MMSE spatial filter, where the desired signal is the transmitted signal, here the desired signal consists of the original transmitted signal modified by desired MIMO channel response H d , which can be interpreted as the "virtual" channel between the G2-encoded signal and the output of the spatial filter.In this case, the desired MIMO channel response H d is These are the subchannels related with the first layer, i.e. the G2 STBC layer.Note that the spatial filter makes no attempt to recover the G2-encoded signal: this will be done exploiting the structure of the STC, which leads to a linear receiver that performs ML detection.The only goal of W is to remove the interference from other layers.The MMSE cost function may be written as The optimal coefficients are found by minimizing the above cost function with respect to W. The solution is given by where are the input covariance matrix and a cross-correlation matrix, respectively.
The coefficients of the MIMO-MMSE spatial filter can be computed after direct least square (LS) estimate of the MIMO channel matrix, which we describe in the following.Note that an estimate of the equivalent channel matrix H d may be extracted from the estimate of H.For instance, in the G2+1 example, H d consists of the first two columns of H.For this sake two training sequences, one for each transmit antenna, are necessary to estimate the MIMO channel H d associated to STBC of the hybrid transmission scheme.Given two training sequences z 11 and z 12 of length Z, the received signal during Z consecutive symbol periods can be expressed as where Thus, the LS channel estimate H d is given by [13] It is worthnoting that the two training sequences necessary to estimate the MIMO channel H d must exhibit good cross-correlation properties and should be optimized to minimize the LS estimation error, which is equivalent to minimizing tr σ 2 v (ZZ H ) −1 [14], for an additive white noise process with variance σ 2 v .The same procedure can be applied to estimate the channels associated to other transmit antennas.
After channel acquisition, the N × N matrix for the MIMO-MMSE spatial filter is found from H d according to the following expression Assuming residual interference at the output of the MIMO-MMSE filter negligible, the output signal can be written as where C = W H d is an equivalent MIMO channel matrix consisting of the original space-time coded channel modified by the coefficients of the MIMO-MMSE filter.This equivalent MIMO channel represents the effective channel that is handled by the space-time decoder and can be interpreted as a virtual channel from the first two transmit antennas of hybrid transmitter to the N outputs of the MIMO-MMSE filter.The term v ′ [k] is a spatially-colored noise vector containing filtered Gaussian noise and residual interference, whose covariance is given by R v ′ v ′ = σ 2 v ′ WW H . Supposing a transmitted sub-sequence s 1 (j) of length K, the space-time decoded signals z(2j + 1) and z(2j + 2), with j = 0, 1, 2, . . ., K, are calculated from the equivalent MIMO channel C by simple linear combining as follows [4] z(2j + 1) where c 1 and c 2 are the first and second columns of C respectively.In (37), we have assumed that the term v ′ , accounting for residual interference plus filtered Gaussian noise is negligible at the input signal y(j) of the modified decoder.However, the optimal decision rule should take into account the covariance matrix R v ′ v ′ .From (36), the conditional probability density function of y[k] can be obtained as (38).
Thus, the maximum likelihood (ML) decoding of the space-time coded signal s 1 [k] is based on the minimization of the following branch metric over all possible codewords of the space-time code used in the transmission.The matrix R −1 v ′ v ′ can be computed adaptively without direct inversion by using the recursive least squares (RLS) algorithm [9].
The approach for channel estimation is the same considered here in the OSIC case when the covariance matrices should be estimated to order the layers.
Figs. 4 to 7 show the architecture of the receivers for all the HMTS.Figures with label (a) show the architecture of the LD receivers for all the HMTS.In these figures we see that all layers are processed in parallel and independently of each other, so no interference cancellation is attempted.Figures with label (b) show the architecture of the SIC receivers for all the HMTS.Clearly, we can see in these figures that the layers are processed successively, in a two stage process in which 1) first a nulling of the interference from the undetected layers is made, then, the output signal goes through a decoder for the STBC used in this layer; 2) finally, the received space-time coded signal corresponding to this layer is regenerated and its impact is cancelled from the received signal.

E. The diversity order of HMTS with interference cancellation
In this section we make some comments about the diversity order of the HMTS regarding the number of degrees of freedom comparing the LD and the SIC algorithms.The claims contained in this section are based on a theorem presented in [11] which we reproduce here. .For 1 ≤ l ≤ L the output of encoder C l is sent using the antennas in group G l and all these transmissions are simultaneous.Let E l , 1 ≤ l ≤ L, denote the average transmit power out of antenna l.Let P 1 denote the probability of error for C 1 using group interference suppression method.Consider another wireless multiple antenna communication system with M 1 transmit and N − M + M 1 receive antennas.At each time k, a block B 1 of bits arrive at the encoder of C 1 .The output of the encoder is sent using the M 1 transmit antennas and all these transmissions are simultaneous.Suppose that the average transmit power out of antenna 1 ≤ l ≤ M 1 is E 1 .Let P 2 denote the probability of error for this system.Then P 1 = P 2 .See the proof in [11].
This theorem shows that, when considering structures of layers and SIC algorithm, the performance of each layer in the HMTS is the same as the performance of the layer subtracting the degrees of freedom needed to cancel the interference of the other layers.For example, consider a MIMO system with 3Tx-3Rx transmitting with scheme G2+1, see Fig. 4. Theorem 1 shows that the error probability of the layer 1 (G2 STBC) that has diversity order of 2(N − 1) is the same of one scheme transmitting with G2 stand-alone, i.e. with two transmit antennas and two receiver antennas.The result of theorem 1 will be used in our next claims.
Due to the nulling-and-cancelling processing of the IC algorithm in the SIC approach, the layers coming after the first one take advantage of the cancelling algorithm and this is translated into a higher diversity order for the next detected layers and the performance of the whole receiver.Considering a general (M Tx-N Rx) MIMO system, it will exist M N communication links between transmitter and receiver.Further, if the antennas are placed far enough from each other, fading occurs independently on each link.If just one of the M N links is not passing through a deep fade, one can sustain communication between transmitter and receiver, with careful system design.Thus, multiple antennas offer M N -fold fade resistance to deep fades.This availability of links is known as the diversity order of multiple antennas, and reflects the fact that they can significantly reduce error rates.
For example, consider the HMTS G2+1 with N receive antennas.Thus, regarding first the linear detector, the first layer will perceive a diversity order of 2(N −1), since this layer has two transmit antennas and it is necessary at least one receiver MIMO spatial filter     antenna to cancel the interference of the second uncoded layer.As the LD detects all the layers at the same time, the second layer has a diversity order of (N − 2), since this layer has one transmit antenna and it is necessary at least two receiver antennas to cancel the interference of the first STBC coded layer.
When the SIC algorithm is used, the first layer will perceive the same diversity order 2(N −1) as in the LD case.However, assuming that all interference is cancelled at the second stage of the SIC, the second layer has now a diversity order of N , since this layer has one transmit antenna and the interference of the first STBC coded layer was already cancelled.This higher diversity order for the second layer explains the better  performance of the SIC compared to LD.Since the bottleneck in performance is due to the uncoded layer, SIC algorithm provides to this layer a higher diversity order, reflecting a benefit in the whole receiver performance.
Considering SIC algorithm, the G2+G2 scheme has a diversity order of 2(N − 2) for the first layer while for the second one we have 2N .For the G3+1 scheme we have a diversity order of 3(N − 1) for the first layer while for the second one we have N .Finally, the G2+1+1 scheme has a diversity order of 2(N − 2) for the first layer and, the second has N − 1 and the last one has N .The performance of the uncoded layers will limit the performance of the whole HMTS, since these layers have no protection at all.A solution was proposed to solve this bottleneck in [16,17].In fact, the diversity of the whole scheme is equal to the diversity of the layer with smallest diversity order.The diversity order of this layer will serve here as comparison parameter.As the G2+G2 scheme has no uncoded layer, since both layers are space-time coded through the G2 scheme, this scheme presents a better result.Following G2+G2, the G3+1 scheme has just one uncoded layer with diversity order of N , while the G2+1+1 has two uncoded layer with diversity order of N − 1 and N , respectively.
Table I, summarizes the multiplexing and diversity orders of the MIMO transmissions schemes.The diversity order is shown for each layer, under both LD and SIC algorithms.When considering a STBC that has just one layer, the diversity order is based on the ML detection.Looking the table we can clearly identify the trade-off between diversity and rate.For example, the STCB G4 can to achieve a diversity order of 4N , but only achieves a symbol of 1/2 symbol pcu.On the other hand, VBLAST achieves a data symbol rate of 4 symbols pcu, but with low diversity order.The HMTS reside between these two extreme points, maximal diversity order (e.g.G4) and maximal multiplexing order (e.g.VBLAST).Clearly, HMTS are inherently flexible structures which can be adapted to the channel conditions, providing more diversity if the channel is in deep fade, or more rate if the channel is experiencing good condition.

V. PERFORMANCE RESULTS
In this section, we present the performance of the proposed transceivers in terms of BER and SER, comparing the HMTS with some conventional MIMO structures.We first compare the IC algorithms performances for the proposed HMTS in order to select the best algorithm for each case.Then, we perform a comparison considering schemes with the same number of transmit and receive antennas.This implies that all schemes have the same potential in achieving the spatial gains (diversity and multiplexing).Then, we compare the performance of HMTS against conventional MIMO structures, but now maintaining the spectral efficiency constant among the several schemes.Finally, we compare the performance of HMTS against LDC.
The performance of the HMTS is evaluated here by means of numerical results from Monte-Carlo simulations.The curves  are plotted against the average Eb/No per receive antenna.Perfect channel estimation is assumed 1 .Unless otherwise noted, all schemes employ binary-phase shift-keying (BPSK) modulation.

A. Comparison of the interference cancellation algorithms
In this section we show the effect of the interference nulling-and-cancelling algorithm, assuming both ordered (OSIC) non-ordered (SIC) successive interference cancellation for the HMTS detection.As benchmark for comparisons, the conventional linear detector was also simulated.We are interested in showing the impact of the IC algorithm in the BER performance of the proposed HMTS.In Figs. 8 to 11, we show the performance of the linear MMSE, SIC and OSIC detectors for different HMTS.As we can see in these figures, the SIC provides a higher diversity than linear detector.This comes from the fact that the combined effect of interference nulling-and-cancelling algorithm provides to the system an additional diversity gain from the one layer to the next to be detected.This leads to an improvement of 1 The degradation due to imperfect channel estimation is negligible if the number of transmit antennas is small [10,15], as in the present case.overall performance.In these figures, we can also see that, OSIC can further provides gain for some schemes, since the error propagation that occurs in SIC may be reduced.This can be observed by the better performance of OSIC as compared to SIC in Figs. 10 and 11.
The similar performance of SIC and OSIC for some HMTS can be explained by our design of SIC.In SIC we used, the first layer to be detected was the one employing a SBTC, which is more robust against fading than the uncoded layers, which transmit with no protection at all.Indeed, with a high probability the layer with highest SNR will be the space-time coded layer, so that OSIC and SIC will likely decode in the same order.This explains the similar result for schemes G2+1 and G3+1.However, in the cases of G2+G2 and G2+1+1, the ordering is not so obvious.In G2+G2, one question is which layer should be detected first.And in G2+1+1 case, which uncoded layer should be detected first.The OSIC provides optimized answers to these questions, which explains why OSIC outperforms SIC in these cases.

B. BER performance of similar complexity schemes
In this section we compare the proposed HMTS with conventional MIMO schemes.Fig. 12 shows BER results comparing the G2+1 HMTS with traditional MIMO schemes for M = N = 3 (G3, H3 and VBLAST), choosing only the curve with the IC algorithm with best performance for each case.We observe that the BER performance of the G2+1 scheme is between those of VBLAST and the STBC schemes.On the other hand, G2+1 achieves a spectral efficiency of 2 symbols pcu, as opposed to 1/2 symbol pcu for G3 and 3/4 symbol pcu for H3.From this results we can conclude that the hybrid scheme G2+1 achieves its objective, i.e. it reaches a higher spectral efficiency than pure STBC schemes, while it has a better BER than a pure VBLAST system.
In Fig. 13, we evaluate the case M = N = 4.In this case, we have more choices of HMTS.Thus, we compare the BER results of the classical MIMO schemes, VBLAST and G4, to the hybrid schemes G3+1, G2+G2 and G2+1+1.Again, only the curve with the IC algorithm with best performance for each case is shown.We can see that more granularity can be achieved when considering four transmit antennas.The best performance is reached by the G4 scheme.This excellent performance is due to its high diversity order, which comes at a cost: G4 transmits only 1/2 symbol pcu.On the other hand, the three HMTS achieve their purpose, having a good BER performance while offering 1.5 symbols pcu in G3+1, 2 symbols pcu in G2+G2 and 3 symbols pcu in G2+1+1.

C. SER performance with similar spectral efficiencies
In this section we compare the performance of some proposed HMTS with STBC and VBLAST schemes.Here, we try to use the same spectral efficiency for all the schemes.To achieve this goal we choose different modulations for each scheme according to the number of data symbols they transmit in each channel use.Our spectral efficiency targets are 3 bps/Hz and 4 bps/Hz.For instance, these spectral efficiencies are achieved by the VBLAST scheme using BPSK and M = N = 3 and M = N = 4, respectively.Table II summarizes our choices for the modulations for each scheme considering 3 bps/Hz, while Table III summarizes our choice targeting 4 bps/Hz.In this section the simpler SIC algorithm was employed whenever its performance was close to OSIC.In Figs. 14 and 15, we plot the SER versus Eb/No for the simulated schemes.We can see that for a fixed spectral efficiency of 3 bps/Hz (Fig. 14) the HMTS have similar performance of pure multiplexing schemes VBLAST, having a better performance in high Eb/No range.The STBC G3 outperforms all the other schemes due to its higher diversity order, since just one receiver antenna is capable of performs ML detection of the transmitted symbols and in this case we are considering three receiver antennas.Thus, in this case we are providing more diversity order to this scheme.
In Fig. 15 we consider a spectral efficiency of 4 bps/Hz as our target, so the modulation cardinality are chosen to reach this goal.The HMTS present better SER performance for almost whole simulated range of Eb/No.This result is an important consequence of the efficient trade-off between spatial diversity and spatial multiplexing achieved by HMTS.In fact, STBC, which are more robust against fading, transmit few symbols per channel use.Thus, to achieve a given spectral efficiency they have to use high-order modulations, which degrades their robustness.We can confirm that affirmation comparing the results of STBC G3 in Figs. 14 and 15.In Fig. 14, since G3 is not designed to provide multiplexing

Scheme
Achievable diversity order LD Achievable diversity order SIC Spectral efficiency η (bps/Hz) VBLAST (4Tx-N Rx)  gain, we achieve this goal through a high cardinality of the modulation, Then, to achieve a spectral efficiency of 3 bps/Hz a 64QAM modulation scheme is necessary and, in this case, we reach a good performance.Once the target is higher, higher cardinality is necessary.Increasing the target in just 1 bps/Hz to 4 bps/Hz leads to G3 increase the cardinality from 64 to 256QAM.The result in the performance degradation of G3 even with ML detection and four receiver antennas is shown in Fig. 15.
As the scheme proposed by Tarokh's et al. in [11] and the HMTS proposed in this work, LDC are also capable of achieving rates higher than one.As an example of comparison of our HMTS and LDC we choose a LDC from [12] designed for two transmit antennas which has rate equal to 2 symbols pcu, referred by Hassibi in [12] just as LDC (31).Thus, we choose two HMTS that have the same rate as this LDC: G2+1 designed for three transmit antennas and G2+G2 designed for four transmit antennas.The simpler SIC algorithm was employed whenever its performance was close to OSIC.In Fig. 16, we can see that the HMTS G2+1 has performance similar to the LDC when N = 3, but in the case of N = 2 LDC Hassibi (31) has the worst performance.HMTS G2+G2 outperforms all HMTS even in this case where the number of receiver antennas is N = 3 showing the good performance of HMTS even when compared to a LDC.

VI. CONCLUSIONS AND PERSPECTIVES
In this paper, we have shown that hybrid MIMO transceiver schemes arise as a solution for the inherent diversity-multiplexing trade-off of MIMO channels.The spectral efficiency of the proposed structures is inherently higher that those of STBC schemes, while their BER/SER performance is better than that of VBLAST structures.We have also shown that jointly interference nulling-and-cancelling algorithm and ML detection of STBC can provide a remarkable improvement in performance compared to the LD approach.Our simulation results show that, for a given spectral efficiency (4bps/Hz) and for the range of Eb/No considered, the HMTS outperform both STBC and VBLAST in terms of SER.
Motivated by current practical limitations in the feasible number of antennas for MIMO systems, we presented HMTS designed for three and four transmit antennas.However the concept of hybrid schemes can be easily extended to large number of antennas.The design and performance analysis of HMTS of higher dimensions is then an interesting topic for future work.The design of hybrid schemes which provide effective coding gain should also be investigated.
A natural continuation of this work resides in devising an adaptive switching algorithm to choose among the various possible MIMO structures presented in this paper, according to the channel characteristics and performance objectives, such as higher link reliability or data throughput.In this way, spatial link adaptation would be enabled by the proposed MIMO structures.

Theorem 1 :
Consider a multiple-antenna wireless communication system with M = L l=1 M l transmit and N ≥ M − M 1 − 1 receive antennas.Let G 1 , G 2 , . . ., G L denote a partition of M transmit antennas into groups M 1 , M 2 , . . ., M L antennas, respectively.Let C = C 1 × C 2 × . . .× C L denote a product space-time encoder.At each time k, a block of B input bits arrive at the encoder of C and these bits are divided into L strings B 1 , B 2 , . . ., B L with B 1 + B 2 + . . .+ B L = B. Every block B l is then encoded by the encoder C l (1 ≤ l ≤ L) HMTS-SIC G2+1 receiver.

TABLE I SUMMARY
OF MIMO TRANSMISSION SCHEMES.