Web Scraper Development for Australian Subcontractors Dataset

Project Overview

Client required a web scraper capable of extracting specific data from a mini-search engine focused on identifying subcontractors traveling between various states/towns within Australia. This scraper's core objective was to navigate through the 'Directory Search' of the website, utilize the 'Search for Carriers by Town' feature, and retrieve a list of subcontractors based on user-defined parameters such as departure state, destination state, and destination town. Beyond this initial list, the scraper also needed to dive deeper into individual subcontractor links to fetch their contact details and other relevant information.

Project Details

Objective

Develop a precise and efficient web scraper to extract data on subcontractors based on user-defined search parameters and compile this data into Excel/CSV datasets.

Key Features

• Two-Level Page Scraper: The scraper was designed to extract data from both the main search results page and individual subcontractor details pages.
• Search Parameter Utilization: Integrated features to input specific search parameters like departure state, destination state, and town to fetch the relevant list of subcontractors.
• Comprehensive Data Extraction: Beyond just the list of subcontractors, the scraper was capable of delving into individual links to retrieve more detailed data like contact information.
• Data Compilation: All scraped data was structured and exported into user-friendly Excel/CSV formats, making it easier for Feilian to utilize and analyze.

Technologies Used

• Web Scraping Tools: Deployed modern web scraping techniques and tools to ensure accurate data extraction.
• Data Compilation: Integrated functionalities to compile the scraped data into Excel/CSV datasets.

Client's Feedback:

Dejan was a star. He completed all the assigned tasks on time and was very patient and thorough with his communication and documentation. Thank you, Dejan!

My Feedback:

This project involved developing a two-level scraper, which was both challenging and rewarding. Collaborating with the client was seamless - he was responsive, knew exactly what he wanted, and was great to work with. I'm always open to assisting with any future projects.