Context: I am interested in the potential to use vector indexing and semantic search for multimodal data. For instance, a retail website may have unstructured text data from the advertising copy on their website, as well as structured text data (in e.g. JSON format) of chat logs between customers and sales agents, and some structured clickstream data.
There are a number of sentence embeddings that claim to be able to make semantic similarity comparisons between data with drastically different surface forms (see OpenAI's ada-002, which outperforms older models on code search, text search, and translation tasks). Theoretically, then, with these models it should be possible to embed and index a multimodal data source. The resulting index should be able to return relevant excerpts from varied sources - text chunks from the website, agent chat logs, or customer website interactions - given a queried topic.
Data: To test whether this sort of system would be possible, I'm looking for an open-source multimodal dataset with three key characteristics:
- At least three data 'modes' - e.g. unstructured text, structured text, structured numeric, clickstream, etc.
- Shared semantic topics - the data sources must focus on shared topics. This is important to prove that semantic comparisons can be made across surface forms.
- Varied surface forms - again, to prove the system works, we need to have data sources that 'look' different, even if the topics they concern are similar.
Region: Any variant of English is OK.
License: Must be usable for commercial purposes (this is a research project, but I work for a private firm hence the requirement).
Format: Very flexible - ideally, unstructured data in .txt files and structured text/clickstream data in .json files.
