Understanding the Difference Between str.contains and str.find in pandas

As a data analyst or scientist, working with text data is an essential part of our job. When it comes to searching for patterns or specific values within a string, two popular methods are str.contains and str.find. In this article, we will delve into the differences between these two methods and explore why they produce different results.

Introduction to str.contains

The str.contains method is used to search for a specified value in a string. It returns a boolean Series denoting the presence (True/1) or absence (False/0) of the value. The method is case-sensitive by default, which means that it treats uppercase and lowercase letters as distinct characters.

import pandas as pd

# Create a sample DataFrame
train = pd.read_csv('https://raw.github.com/mattdelhey/kaggle-titanic/master/Data/train.csv')

# Check if the string 'Mr.' contains any value
print(train.name.str.contains('Mr.').sum())

Introduction to str.find

The str.find method, on the other hand, is used to find the index of the first occurrence of a specified value in a string. It returns an integer representing the position of the value if found; otherwise, it returns -1.

import pandas as pd

# Create a sample DataFrame
train = pd.read_csv('https://raw.github.com/mattdelhey/kaggle-titanic/master/Data/train.csv')

# Find the index of the first occurrence of 'Mr.'
print((train.name.str.find('Mr.') > 0).sum())

The Difference Between str.contains and str.find

The main difference between str.contains and str.find lies in their behavior when dealing with regular expressions. By default, str.contains treats the entire string as a single entity, whereas str.find only searches for the specified value within the string.

In our example, we can see that the code snippet using str.contains returns 647 results, while the code snippet using str.find returns 517 results. This discrepancy is due to the way these two methods handle regular expressions.

Escape Special Characters

One way to resolve this issue is to escape special characters within the specified value. In the case of str.contains, we need to escape the period (.) by prefixing it with a backslash (\).

import pandas as pd

# Create a sample DataFrame
train = pd.read_csv('https://raw.github.com/mattdelhey/kaggle-titanic/master/Data/train.csv')

# Check if the string 'Mr.' contains any value, escaping special characters
print(train.name.str.contains('Mr\.").sum())

Disabling Regular Expression Matching

Alternatively, we can disable regular expression matching by setting the regex parameter to False.

import pandas as pd

# Create a sample DataFrame
train = pd.read_csv('https://raw.github.com/mattdelhey/kaggle-titanic/master/Data/train.csv')

# Check if the string 'Mr.' contains any value, disabling regular expression matching
print(train.name.str.contains('Mr.', regex=False).sum())

Handling Non-Matching Results

When using str.contains, we can handle non-matching results by concatenating two Series: one for values that contain the specified string and another for values that do not.

import pandas as pd

# Create a sample DataFrame
train = pd.read_csv('https://raw.github.com/mattdelhey/kaggle-titanic/master/Data/train.csv')

# Concatenate Series to handle non-matching results
a = train.loc[train.name.str.contains('Mr.'), 'name']
b = train.loc[(train.name.str.find('Mr.') > 0), 'name']

c = pd.concat([a, b], axis=1, keys=('contains','find'))

# Filter out rows with missing values
c = c[c.isnull().any(axis=1)]

print(c)

Conclusion

In conclusion, while str.contains and str.find seem similar at first glance, they have distinct differences in terms of behavior when dealing with regular expressions. By understanding these differences and using the appropriate methods, we can accurately search for patterns or specific values within our text data.

Whether you choose to escape special characters, disable regular expression matching, or handle non-matching results, there are numerous ways to resolve discrepancies between str.contains and str.find. In this article, we have explored some of these options in-depth, providing you with a comprehensive understanding of the differences between these two popular pandas methods.

Last modified on 2023-05-18