Data types
We can get information about the columns in the DataFrame by using the .info() method.
refugee_df.info()
This report tells us how many non-null, or non-blank, values are in each column, as well as what type of data is in each column.
Pandas uses a different lexicon to describe data types from those we learned in our intro to Python curriculum. Below is a table that explains what each data type means:
| Pandas data types | Python data types | Usage |
| object | String or mixed | Text or mixed numeric and non-numeric values |
| float64 | float | Floating point numbers |
| int64 | integer | Integer numbers |
| datetime64 | NA | Date and time values |
Converting data types
Keeping this in mind, it looks as though the data type for the year column is a “int64” instead of being “datetime64.”
First, let’s define a new variable for the year columns in “refugee_df” DataFrame.
refugee_int = refugee_df['year']Next, we can run the command below to convert the data type:
refugee_df['year'] = pd.to_datetime(refugee_int, format='%Y')This command says: for the “year” column in the “refugee_df” DataFrame, use the “to_datetime” method in the Pandas library to convert the values in the “year” column in the “refugee_df” DataFrame, as defined by the variable “refugee_int”, to datetime data types, using the format “%Y” for just year (as opposed to %Y%m%d, which would also include the month and day).
We can then check to see if the data type was properly converted using the .dtypes object, which is similar to the .info() method, except it only provides information on data types.
refugee_df.dtypes
As we can see, the data in the “year” column was successfully transformed into the datetime64 data type.
Check for duplicate rows
As part of our data cleaning process, we want to check for duplicate rows. We can do this by using the .duplicated() method inside a filter to isolate only the rows in the DataFrame that are exact duplicates. Filtering data by certain values is similar to selecting columns.
refugee_df[refugee_df.duplicated(keep=False)]
Looks like we have a few duplicate rows in our dataset.
To remove those duplicates, we can use the .drop_duplicates() method to drop duplicates from the DataFrame and select to keep the first instance of the duplicate or the last instance:
refugee_df = refugee_df.drop_duplicates(keep='first')We can check to see if the command got rid of the duplicate rows by running the .duplicated() method again:
refugee_df[refugee_df.duplicated(keep=False)]
Great news! We successfully removed our duplicate rows!
Terms used in lesson:
.info(): The .info() method in Pandas tells us how many non-null, or non-blank, values are in each column, as well as what type of data is in each column.
.dtypes: The .dtypes object in Pandas tells us what type of data is in each column.
.duplicated(): The .duplicated() method in Pandas checks for duplicate rows.
.drop_duplicates(): The .drop_duplicates() method in Pandas drops duplicate rows.
