Data Processing with pandas¶

pandas¶

pandas is a popular Python package with excellent functionality for data analysis tasks.

DataFrames¶

A DataFrame is a pandas data structure, much like a table:

• values are held in rows and columns
• each row and column has a label

Typically, rows are numerically labelled, starting at zero, while columns have names in strings.

However, a DataFrame can have strings for row labels and they don't have to be unique.

Example repl¶

Let's jump in with an example. Fork the repl, remembering as always to add your credentials to a .env file.

!!! info The walkthrough will highlight a handful of pandas techniques used for creating the Seeder app, but if you would like a more complete understanding of pandas and these techniques, please consult the excellent documentation.

Code walkthrough¶

Not much new to see in main.py; we've just substituted use of the .get_recommmendations() method for the .get_item() method we used previously to get individual track data:

recs = s.get_recommendations(track_id)
for track in recs:  
    ...  
    print(f'{track_name} - {track_artist}')

• .get_recommendataions() returns a list containing data for several tracks, over which we iterate to print the track_name and track_artist
• pandas has also been imported at the top of the script, and given the alias of pd

In spotter.py we'll find .get_recommendations() at the bottom, but also notice a small modification to .get_item():

def get_item(self, category, item_id=None, params=None):
    ...
    resp = requests.get(f'{self.api_root}{category}/{item_id}', params=params, headers=self.headers)

• the get_item() method no longer requires an item_id, and can also accept a params argument

def get_recommendations(self, seed, number=6):
    recs = self.get_item('recommendations', params={'seed_tracks': seed, 'limit': 100})

• we use .get_item() to fetch some recommendations
  • the track_id defined in main.py is the value for the seed_tracks parameter
  • we set a limit of 100 items to be returned

Creating a DataFrame¶

• the value for the tracks key in recs was a list of 100 dictionaries, each one relating to a given track
• the pd.DataFrame() function has created a DataFrame from the list, with:
  • the data for each track being within a single row
  • the column headings are the keys from the track dictionaries
• the .head() method allows us to show just the first few records

Using .apply() and lambda functions¶

In this part of the code, we're going to extract the number of tracks we tracks we're looking for (which is 6 by default), but only take those which are available in as many markets as possible.

We'll do this by:

1. Adding a column which contains number of markets listed in available_markets for each track
2. Filtering the DataFrame to include only those with the max() number found
3. Using the .head() method to fetch the first 6 tracks from the remaining tracks

• a new column availability has been created (scroll right to see it)
• the .apply() method was used on the available_markets column, with a lambda function used as its argument
• this lambda function means "take the original value as x, and return len(x), i.e. the length of the list"

Lambda functions are also known as anonymous functions, and tend to be found where the logic is only required for a short period of time, i.e. it will not be re-used.

Selecting data¶

• this can be read as "return the df [where the ['availablility] column in that df is equal to == the .max() value in that same column ]"
• ...and the return the .head() of the result, containing the number of values specified
• we then used .head() again to display the first two rows of the result

• here we use .apply() again, this time incorporating our .get_item() method and passing the value in the id column as one of the arguments to get audio-features data for each track
• we can see a subset of columns of the tracks DataFrame by providing a list of columns in [square brackets ]

reset_index()¶

• notice in our previous outputs that the row indexes have been retained from the original DataFrame (and aren't continuous or starting from zero)
• it will be helpful for us later on to reset these so that they are; the drop=True parameter means that our original index values won't be stored (drop=False would have resulted in the creation of a new column containing them)

Unpacking JSON¶

• the json_normalize() function is used here to unpack the dictionaries returned by our .get_item() method calls requesting audio-features for each track

Concatenation¶

• we have concatenated (joined together) the two DataFrames
• axis=1 means that we do so in a column-wise manner
• the row indices are used to match the data from each DataFrame
• the .shape attribute of a DataFrame contains the number of (rows, columns)

The pandas .join() and merge() methods are useful for more complex SQL-like joins using specific columns as keys.

• .duplicated() is a method which identifies any repeat instances of column names

Location with .loc¶

• .loc is used to locate items in our DataFrame
• the code before the , refers to the rows; here : means 'all rows'
• the code after the , refers to the columns; ~ means 'not' or 'opposite'
• .copy() makes a new object so that subsequent operations on tracks_all can be made without tracks_full being affected (to save memory, pandas may not otherwise store such a subset as a distinct object)
• we now have four fewer columns; the duplicates have been removed

Multiple artist names¶

Sometimes our tracks will have multplie artists, and it would be nice to be able to use all of their names in our app.

Here we again use .apply() with a lambda function, incorporating the .join() method and a list comprehension to amalgamate the name values for all entries in artists:

Data structure conversion¶

• here we use the .to_dict() DataFrame method, with the orient="records" parameter, to return a list of dictionaries, each one containing data about a given track
• each of these dictionaries has the same structure (except with the additional audio-features and our new artist_string key:value pairs) as seen previously when calling .json() on a Response object returned from a get_item() request