Dimensionality Reduction in Embedding Spaces: A Comparative Analysis of DCT and SVD for RAG Systems

Kevin Scott and Ethan Davis — Sat, 20 Jan 2024 00:00:00 GMT

Abstract

We present a methodological framework for evaluating dimensionality reduction techniques in dense embedding spaces for Retrieval-Augmented Generation (RAG) systems. Specifically, we examine Discrete Cosine Transform (DCT) and Singular Value Decomposition (SVD) applied to the column space of embedding matrices, followed by quantization. Our evaluation protocol centers on retrieval consistency - measuring whether reduced representations retrieve the same contextual chunks as full-dimensional embeddings. This work was conducted in collaboration with Ethan Davis at SAS Institute Inc.

Introduction

Dense embeddings from transformer-based models typically operate in high-dimensional spaces (d ∈ {384, 768, 1024, 1536}). While these representations capture rich semantic information, their dimensionality poses computational and storage challenges. This work examines column-space reduction techniques that aim to preserve retrieval quality in RAG pipelines.

Theoretical Framework

Column Space Reduction

Given an embedding matrix where represents the number of documents and the embedding dimension, we seek a transformation where .

Discrete Cosine Transform (DCT)

The DCT provides an orthogonal transformation that concentrates signal energy in low-frequency components:

where is the DCT matrix. We retain the first columns of , exploiting the energy compaction property commonly observed in natural signals.

Singular Value Decomposition (SVD)

We apply SVD directly to the embedding matrix for dimensionality reduction:

Compute SVD:
Truncate to components:
Project embeddings:

where contains the first right singular vectors, effectively projecting from to .

Quantization Strategies

Post-reduction quantization further compresses representations:

Scalar Quantization

Map continuous values to discrete levels
B-bit quantization: 2^B levels
Consider Lloyd-Max quantizer for optimal level placement based on distribution

Vector Quantization

k-means clustering in reduced space
Codebook size determines compression ratio
Product quantization for large-scale applications

Proposed Evaluation Methodology

Retrieval Consistency Metric

For query , let: - = top- retrieved chunks using full embeddings - = top- retrieved chunks using reduced embeddings

We define retrieval consistency as:

Experimental Protocol

Corpus Preparation: Segment documents into semantically coherent chunks
Embedding Generation: Apply pre-trained encoder (e.g., BERT, Sentence-BERT, E5)
Dimensionality Reduction:
- For SVD: Compute on training subset, apply to full corpus
- For DCT: Direct application of transformation matrix
Quantization: Apply scalar or vector quantization to reduced embeddings
Evaluation:
- Sample diverse query set
- Compare retrieval sets between full and reduced representations
- Measure RC@k for k ∈ {1, 5, 10, 20, 50}
- Analyze distribution of retrieval rank changes

Additional Metrics

Beyond retrieval consistency, consider: - Semantic Similarity Preservation: Correlation between cosine similarities in original vs. reduced space - Computational Efficiency: Indexing time, query latency, memory footprint - Reconstruction Error: where is the pseudo-inverse

Theoretical Considerations

Information-Theoretic Perspective

The fundamental question: what is the intrinsic dimensionality of semantic embeddings? Rate-distortion theory suggests embeddings contain redundancy that can be exploited.

DCT vs SVD: Key Differences

SVD: - Data-dependent transformation - Optimal linear dimensionality reduction (in Frobenius norm) - Requires singular value decomposition - Captures global variance structure

DCT: - Data-independent transformation - Assumes local smoothness in embedding space - computation via FFT - Natural frequency interpretation

Implementation Considerations

Computational Complexity

SVD: for full decomposition
DCT: via FFT implementation
Quantization: for scalar, for -means vector quantization

Numerical Stability

When implementing SVD on high-dimensional embeddings:

# Direct SVD on embedding matrix
U, S, Vt = np.linalg.svd(embeddings, full_matrices=False)
projection_matrix = Vt[:k].T
reduced_embeddings = embeddings @ projection_matrix

Open Research Questions

Optimal Dimensionality Selection: How to determine k without extensive empirical search?
Query-Dependent Reduction: Can we adaptively select dimensions based on query characteristics?
Non-linear Alternatives: Would manifold learning techniques preserve retrieval quality better?
End-to-End Learning: Can we train embeddings aware of downstream compression?

Conclusion

This work outlines a rigorous framework for evaluating dimensionality reduction techniques in RAG systems. The trade-off between computational efficiency and retrieval quality remains an active area of research. Future work should focus on establishing theoretical guarantees and developing adaptive compression strategies.

Acknowledgments

This research was conducted in collaboration with Ethan Davis at SAS Institute Inc. We thank the SAS Institute for computational resources and support.

References

Johnson, W. B., & Lindenstrauss, J. (1984). Extensions of Lipschitz mappings into a Hilbert space.
Jegou, H., Douze, M., & Schmid, C. (2011). Product quantization for nearest neighbor search.
Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks.

This methodology paper outlines ongoing research. Implementations and empirical results will be shared upon completion of experiments.

Kevin Scott — AI Systems & Performance Engineering