{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Similarity Scoring Demo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Zebra finch song is learned by imitating the song of an adult tutor. We can measure pupil bird's song learning success by comparing its song to its tutor's song, with higher similarity indicating better learning. \n", "\n", "AVN's similarity scoring module works by calculating 8-dimensional representations (or *embeddings*) of syllables from different birds, then calculating the distance between the two birds' syllable distributions. The embeddings are generated by feeding spectrograms of segmented syllables to a trained neural network that embeds similar syllables close together and dissimilar syllables far apart. We've shown that this model generalizes to zebra finches across multiple colonies and recording setups, so there should be no need for you to retrain the model. Just feed it your spectrograms and get scoring! \n", "\n", "The more syllables from each bird, the more reliable the scores you will obtain. Generally, we recommend at least 1000 total syllables from each bird (about 100 bouts), but more is better." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before you can use this similarity scoring module, you must install some extra dependencies. We strongly recommend using virtual environments when working with any python code, including AVN. See [here](https://docs.anaconda.com/free/navigator/getting-started/) for more information. \n", "\n", "In the virtual environment where you previously installed AVN, install pytorch according to your system requirements, as explained [here](https://pytorch.org/get-started/locally/). \n", "\n", "Next, you will need to install the package that we use for EMD calculations. That can be done by running this command from a command prompt with your virtual environment activated: \n", "```\n", "pip install -e git+https://github.com/theresekoch/pyemd.git#egg=emd\n", "```\n", "Once you've done that you should be all set! If you run into any issues installing these dependencies, please reach out via [email](mailto:therese.koch1@gmail.com) or through github. " ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "c:\\Users\\tkoch\\anaconda3\\envs\\fresh_avn_dev\\lib\\site-packages\\scipy\\__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3\n", " warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n" ] } ], "source": [ "import avn.similarity as similarity\n", "import pandas as pd\n", "\n", "from sklearn.decomposition import PCA\n", "import matplotlib.pyplot as plt\n", "import numpy as np" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Prepare Spectrograms " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before we can calculate the embedding coordinates, we need to generate spectrograms of each segmented syllable in a way that is compatible with our embedding model. This is done by calling the function `similarity.prep_spects(Bird_ID, segmentations, song_folder_path, out_dir)`, where: \n", "\n", "- `Bird_ID` is the ID of the subject bird, \n", "- `segmentations` is a pandas dataframe of syllable segmentations, with one row per syllable, and columns called :\n", " - 'files' : the name of the .wav file in which a syllable is found, \n", " - 'onsets' : the onset of the syllable within the .wav file in seconds, and \n", " - 'offsets': the offset of the syllable within the .wav file in seconds. \n", " - We recommend using [WhisperSeg](https://github.com/nianlonggu/WhisperSeg) to automatically segment song syllables and generate this type of table.\n", "- `song_folder_path` is the path to the folder containing all the wav files from `segmentations`. \n", "- `out_dir` is the path to the folder where you want the generated spectrograms to be saved. \n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
| \n", " | files | \n", "onsets | \n", "offsets | \n", "labels | \n", "
|---|---|---|---|---|
| 0 | \n", "G402_43362.23322048_9_19_6_28_42.wav | \n", "0.166009 | \n", "0.225170 | \n", "i | \n", "
| 1 | \n", "G402_43362.23322048_9_19_6_28_42.wav | \n", "0.321043 | \n", "0.385828 | \n", "i | \n", "
| 2 | \n", "G402_43362.23322048_9_19_6_28_42.wav | \n", "0.570295 | \n", "0.626757 | \n", "i | \n", "
| 3 | \n", "G402_43362.23322048_9_19_6_28_42.wav | \n", "0.690363 | \n", "0.751837 | \n", "i | \n", "
| 4 | \n", "G402_43362.23322048_9_19_6_28_42.wav | \n", "0.828390 | \n", "0.898912 | \n", "i | \n", "