Representations of complicated spaces by vectors which preserve semantic information.
Warning: this is not my current area, but it is a rapidly moving one.
Treat notes here with caution; many are outdated.
Modernised for transformer era
Vicki Boykis’ series What are Embeddings?
Technical survey: Kleyko et al. (2022) cites back to the year 2000.
Feature construction for inconvenient data;
made famous by word embeddings such as
being surprisingly semantic.
Note that word2vec has a complex relationship to its documentation.
We map categorical variables in a function approximation problem into Euclidean spaces, which are the entity embeddings of the categorical variables. The mapping is learned by a neural network during the standard supervised training process. Entity embedding not only reduces memory usage and speeds up neural networks compared with one-hot encoding, but more importantly by mapping similar values close to each other in the embedding space it reveals the intrinsic properties of the categorical variables. We applied it successfully in a recent Kaggle competition and were able to reach the third position with relative simple features. We further demonstrate in this paper that entity embedding helps the neural network to generalize better when the data is sparse and statistics is unknown. Thus it is especially useful for datasets with lots of high cardinality features, where other methods tend to overfit. We also demonstrate that the embeddings obtained from the trained neural network boost the performance of all tested machine learning methods considerably when used as the input features instead. As entity embedding defines a distance measure for categorical variables it can be used for visualizing categorical data and for data clustering.
Rutger Ruizendaal has a tutorial on learning embedding layers
Embedding vector databases
Related: learnable indices.
LangChain docs maintains a confusing but current list of backing sores: VectorStores
Built on top of popular vector search libraries including Faiss, Annoy, HNSW, and more, Milvus was designed for similarity search on dense vector datasets containing millions, billions, or even trillions of vectors. Before proceeding, familiarize yourself with the basic principles of embedding retrieval.
Milvus also supports data sharding, data persistence, streaming data ingestion, hybrid search between vector and scalar data, time travel, and many other advanced functions. The platform offers performance on demand and can be optimized to suit any embedding retrieval scenario. We recommend deploying Milvus using Kubernetes for optimal availability and elasticity.
Milvus adopts a shared-storage architecture featuring storage and computing disaggregation and horizontal scalability for its computing nodes. Following the principle of data plane and control plane disaggregation, Milvus comprises four layers: access layer, coordinator service, worker node, and storage. These layers are mutually independent when it comes to scaling or disaster recovery.
Milvus Lite is a simplified alternative to Milvus that offers so many advantages and benefits.
- You can integrate it into your Python application without adding extra weight.
- It is self-contained and does not require any other dependencies, thanks to the standalone Milvus' ability to work with embedded Etcd and local storage.
- You can import it as a Python library and use it as a command-line interface (CLI)-based standalone server.
- It works smoothly with Google Colab and Jupyter Notebook.
- You can safely migrate your work and write code to other Milvus instances (standalone, clustered, and fully-managed versions) without any risk of losing data.
This tool provides an efficient implementation of the continuous bag-of-words and skip-gram architectures for computing vector representations of words. These representations can be subsequently used in many natural language processing applications and for further research.”
fastText is a library for efficient learning of word representations and sentence classification.