Common index but not contract

[chiamin]

It would be useful to have an operation for $C_i = A_i B_i$ or more general $C_i = \sum_j A_{ij} B_{ij}$. We may want to add this method in the future.

[ianmccul]

The first one is a Hadamard product, doesn't exist for symmetric tensors, but as an operation on a generic tensor it certainly has uses. It should already be possible somehow - I noticed that operation is implemented in src/backend/linalg_internal_cpu/Mul_internal.cpp. I assume it is just Mul(A,B), defined in include/linalg.hpp lines 406-430 ?

The second one is the column sum of $AB^T$, and there isn't going to be much gain by doing it in one operation as the contraction will be the slow part. So I'd express this as $T_{ik} = \sum_j A_{ij} B_{jk}$ followed by $C_i = \sum_k A_{ik}$.
The summation over one (or more) indices is also an operation that doesn't apply to symmetric tensors, but would be useful for a generic tensor. I haven't seen that it is implemented in Cytnx, but you can get the same effect by a contraction with ones(n).

[ianmccul]

Or is it is the diagonal part of $A_{ik}$, not the column sum?

[chiamin]

What I had in mind is something like the / operator between two tensors in ITensor C++ version. See their documentation. (https://itensor.org/docs.cgi?vers=cppv3&page=classes/itensor) I copy their explanation as follows which may be more clear.

Non-contracting product (has no relationship to division). A / B creates a new tensor out of A and B by "merging" any common indices according to the rule Rijk = Aik Bjk (no sum over k). (Here i, j, and k could be individual indices or represent groups of indices.)

It should work also for symmetric tensors. For Cytnx, I was thinking of integrating this method into Contract and Network by specifying the indices that are common but remain in the outcome tensor, but I didn't carefully think about the interface. We can also make another method.

An application of the method is as follows. In QTT a function $f$ is an MPS. When computing $f^2$, we need to compute $T_{i,l1,r1,l2,r2}=A_{i,l1,r1} A_{i,l2,r2}$ which requires this method. I also used it from time to time previously with symmetric tensors but hard to explain...

[ianmccul]

Right, QTT has this concept of 'lifting' a MPS into an MPO, which I think works by regarding the MPS as an MPO with those elements on the diagonal? $A^s \rightarrow W^{s's} = A^s \delta_{s's}$. If the MPO is in sparse format (with respect to the physical degrees of freedom), then you can just construct $W^{s's}$, and it will be just as fast as some special multiplication operator on $A^s$. But I guess it is nice to support both methods.

How does this work with symmetric tensors? If you put quantum number labels on $s$, you surely can't preserve the same label on the diagonal part of $W^{s's}$?

[chiamin]

Right, QTT has this concept of 'lifting' a MPS into an MPO, which I think works by regarding the MPS as an MPO with those elements on the diagonal? A s → W s ′ s = A s δ s ′ s .

Yes exactly.

If the MPO is in sparse format (with respect to the physical degrees of freedom), then you can just construct W s ′ s , and it will be just as fast as some special multiplication operator on A s . But I guess it is nice to support both methods.

Oh I didn't think of spares tensor can do the same job, but you are right.

How does this work with symmetric tensors? If you put quantum number labels on s , you surely can't preserve the same label on the diagonal part of W s ′ s ?

I don't quite the problem here. What I had in mind is the abelian quantum numbers (which is the only type that ITensor supports), where I think of a symmetric tensor as a block-diagonalized tensor, and can be thought as a dense tensor with many zeros. In this case, this operation should be well-defined?

[ianmccul]

I don't quite the problem here. What I had in mind is the abelian quantum numbers (which is the only type that ITensor supports), where I think of a symmetric tensor as a block-diagonalized tensor, and can be thought as a dense tensor with many zeros. In this case, this operation should be well-defined?

No, its not well-defined. I mean, you can write down those matrix elements, but the result isn't a symmetry-preserving tensor. By definition, a tensor is symmetric if you do a symmetry transformation $U = \exp (i n \theta)$, and it satisfies

$X^\prime_{i\prime j \prime k \prime \ldots } = U_{i\prime i} U_{j\prime j} U_{k\prime k} \ldots \quad X_{ijk \ldots}$

(assuming all of the indices are kets -- for bra indices apply $U^\dagger$ instead)

Normal tensor contraction satisfies this, because the contraction gives a $U^\dagger U = I$ and it drops out. But Hadamard product and other operations that do not respect the Einstein summation rules don't satisfy this. So the result is basis-dependent, in the sense that if you apply a (symmetry preserving) unitary transformation to the input tensors, you cannot shift that unitary and write an equivalent operation where the unitary is acting on the output tensor.

In other words, if you changed basis of your physical Hilbert space (which means rotating by a quantum number-dependent phase factor), then this shouldn't change any physical expectation values - the $U$ transformation of the basis cancels out with a corresponding $U^\dagger$ acting on the operator. But a Hadamard product and similar 'contractions' don't have this structure.

Another way of seeing this is that you can express these as normal contractions using the 3-index 'join tensor', that takes a two index object into a 1-index object $\delta_i^{jk} = 1$ if $i=j=k$, and $0$ otherwise. Unlike the Kronecker delta, though, this isn't a symmetric tensor because applying a unitary to each leg doesn't give back the same tensor. And you can't write it as a symmetry-preserving tensor, since it cannot preserve quantum numbers $q_i = q_j + q_k$.

QTT doesn't have this symmetry structure, since it uses a very specific basis, mapping the domain $x$ into binary digits, and many operations on a QTT depend on that choice. Imagine rotating the QTT into a basis of $\{0 \rightarrow \ket{0}, 1 \rightarrow -\ket{1}\}$. This is equivalent to applying $(-1)^N$ operator to a $U(1)$ MPS. This will completely mess up the matrix elements of $f^2$ -- the result won't be the same as constructing $f^2$ in the original basis and then rotating $\ket{1} \rightarrow -\ket{1}$; the squaring operation doesn't commute with the symmetry rotation.

[chiamin]

Yes, you are right! Thanks for the explanation. Now I recalled what I did with symmetric tensors. I used tensors with a mixture of symmetric and non-symmetric indices, and the index to be "merged" must have no quantum number. This index is used as a label for symmetric tensors.

With QTT I don't use any quantum number, so the method for non-symmetric tensors will do the job.

[ianmccul]

Yeah, another example is if you have a simple array of tensors all of the same shape, say a vector<Tensor> in C++. In some sense that is a tensor itself, but the array index doesn't have a quantum number index (or you could consider it to be zero). It is probably worthwhile storing this as an actual tensor, because then it can be stored in bigger blocks, rather than $N$ individual tensors. I've been thinking about data structures like tensors of tensors, and how to coalesce it into a single data structure. That is how I store MPO's, as a matrix of local operators, but I don't attempt to optimize the structure. Well, I think the structure I use is already basically optimal for most ordinary MPS operations, but there are situations where it could help though. I would want to use a different structure with QTT, and change the order of the indices. That requires rearranging the fusion tree, in the symmetric case.