Rearranging Table after PDF Extraction with Tabula

In this article, we will delve into the process of rearranging tables extracted from PDFs using the Tabula library in Python. We will explore a common issue that arises when dealing with table extraction and provide a solution to tackle it.

Table Extraction with Tabula

Tabula is a powerful library used for extracting tables from PDF files. It can handle various types of tables, including those with multiple columns and rows. The library uses optical character recognition (OCR) technology to identify the table structure in the PDF.

However, one common issue that arises when using Tabula is when a row contains wrapped text, causing the library to split it into two separate rows. In such cases, the first row may contain most of the data, while the second row is filled with “None” values for missing information.

Solving the Issue

To solve this issue, we need to modify our approach to how we handle table extraction and cleaning.

Step 1: Understanding the Problem

The problem arises because Tabula assumes that a row contains only one cell. When it encounters wrapped text, it splits the row into two separate rows, each with most of the data in the first row and “None” values in the second row.

To tackle this issue, we need to identify the correct row and merge its contents with the previous row.

Step 2: Modifying the `CleanRunResults` Function

The provided function, CleanRunResults, appears to be on the right track. However, it needs some modifications to correctly handle rows with wrapped text.

def CleanRunResults(df):
    for row in range(len(df)-1, -1, -1):
        NoArea = pd.isnull(df['Area'].iloc[row])
        NoShape = pd.isnull(df['Shape'].iloc[row])
        YesType = pd.notnull(df['House_Type'].iloc[row])  # Fix: Corrected column name
        PrevRow = row - 1
        if NoArea and NoShape and YesType:
            # Check if the current row contains wrapped text
            if df['House_Type'].iloc[row].str.contains(r'\n').any():
                # Merge the contents of the current row with the previous row
                df['House_Type'].iloc[PrevRow] += ' ' + df['House_Type'].iloc[row]
                df.drop(row, inplace=True)
    df.dropna(subset=['Shape', 'Area'], how='all', inplace=True)
    df = df[['House_Type', 'Shape', 'Area']]
    return(df)

In this modified version of the function:

We first check if the current row contains wrapped text using df['House_Type'].iloc[row].str.contains(r'\n').any().
If it does, we merge the contents of the current row with the previous row using string concatenation.
Finally, we drop the current row from the dataframe to prevent duplicate rows.

Step 3: Understanding the Code

In the modified function:

df['House_Type'].iloc[row].str.contains(r'\n').any() checks if any row in the ‘House_Type’ column contains a newline character (\n). The \n is used to indicate wrapped text.
df['House_Type'].iloc[PrevRow] += ' ' + df['House_Type'].iloc[row] merges the contents of the current row with the previous row. The spaces are added between the two rows’ texts to create a single, continuous line.

By following these steps and modifying the CleanRunResults function accordingly, we can effectively tackle the issue of table rearrangement after PDF extraction using Tabula.

Example Use Cases

Example 1: Extracting Tables from PDFs

import pandas as pd
from tabula import read_pdf

# Read a PDF file
df = read_pdf('example.pdf', pages='1-3')

# Print the extracted table
print(df)

Example 2: Rearranging Table Rows with Wrapped Text

import pandas as pd

# Create a sample dataframe with wrapped text
data = {'House_Type': ['Blue House\nwith multiple lines', 'Red house'],
        'Area': [3456, 2345],
        'Shape': ['circle', 'square']}
df = pd.DataFrame(data)

# Apply the CleanRunResults function to rearrange rows
df = CleanRunResults(df)

# Print the rearranged table
print(df)

Conclusion

In this article, we explored a common issue that arises when extracting tables from PDFs using Tabula. We modified the CleanRunResults function to correctly handle rows with wrapped text and provide an efficient solution for rearranging table rows.

By following these steps and applying the provided code examples, you can effectively tackle the challenge of table extraction and cleaning in Python.

Last modified on 2024-07-02