Metadata Parser (Scraper) Python Script

Web Scraper for Website Metadata

Description: This web scraping tool efficiently extracts metadata details such as Title, Meta Description, and Canonical URL from a list of given web URLs. Built using Python, the script leverages the BeautifulSoup and Requests libraries to fetch and parse the webpage data. It then exports the extracted details into an Excel file for easy accessibility and analysis.

Key Features:

• Batch Processing: Process multiple URLs concurrently.
• Error Handling: The script handles potential request errors and ensures smooth processing.
• Dynamic User-Agent Rotation: Reduces the chances of getting blocked by websites.
• Excel Export: Outputs the scraped data into an Excel file.
• Progress Logging: Provides real-time logging of the processing status.

Installation Guide:

1. Prerequisites: Ensure you have Python 3.x installed. You can verify the installation with:

   python --version

2. Install Required Libraries: Navigate to the directory containing the script and run:

   pip install requests beautifulsoup4 openpyxl

3. Download the Script: Save the provided Python script in your preferred directory.

Usage Guide:

1. Prepare the Input File:
   • Create a CSV file named url.csv in the same directory as the script.
   • Add all the URLs you want to scrape, one URL per line.
2. Run the Script: Navigate to the directory containing the script in your terminal or command prompt and run:

   python meta_parser.py

   
   Replace <script_name>.py with the name you've saved the script as.
3. Optional Arguments:
   • --input: Specify a different input file name. Default is url.csv.
   • --output: Specify a different output Excel file name. Default is result.xlsx.
   Example usage with optional arguments:

   python meta_parser.py --input myurls.csv --output metadata_results.xlsx

4. Review the Results: Once the script finishes processing, you can find the result.xlsx file (or your specified output filename) in the same directory. This Excel file will contain the scraped Title, Meta Description, and Canonical URL for each input URL.
5. Logs: While the script runs, it will display logs indicating the progress and any potential errors encountered. This will help you monitor the status and troubleshoot if required.

Notes:

• Websites may change their structure over time. If you find the script is not extracting data correctly, it may be due to structural changes in the target websites. In such cases, minor adjustments to the script might be necessary.
• Avoid overloading a website with requests in a short period. Always respect robots.txt and terms of service of websites.

Support:

For any issues or enhancement requests, please contact the developer or consult the documentation. Always ensure you're using the latest version of the script for optimal performance.