Web Scraping of unstructured healthcare data from multiple web threads using NLP and RPA

Challenges.

The team at HitechDigital studied the processes to understand the scope of work, technology to be used, and the workflow to be designed. Following project requirements increased the challenges of the process:

• Extract unstructured data in form of paragraph or text, which had no repetitive pattern
• Identify, flag and skip comments removed by moderators – not to be collected
• Expand sub replies and collapsed comments according to the hierarchy to collect/scrape data
• Skip conversations that broke the continuity of discussion thread and led to another page altogether
• Omit extraneous data such as time stamps, points, etc., which increased project complexity

Solution.

Designed an ongoing process of web scraping and unstructured data extraction using NLP and RPA to collect data from healthcare posts and discussion on Reddit subgroups. The automated data management workflow ensured that collected data upon reaching the final lag, would trigger a macro to move the data through a predefined quality and profiling process.

Approach.

Manual data extraction requires complex workflows and significant hand-coding to extract, cleanse, and validate unstructured data. So, data professionals at HitechDigital started off by deriving a smarter, easier way to automate unstructured data extraction workflows.

Implementation:

• The email/FTP/folder integration feature of the automated workflow was leveraged to receive sets of sub-Reddit pages and pre-defined instruction for output.
• Programmable bots and scripts were used to scrape and tag discussion threads in pre-defined hierarchical sequence like discussion title, description, username and replies.
• The execution order was:
  • Pick a topic and collect all the discussion links and its count from healthcare sub-Reddit pages
  • Go through each discussion > Sort by OLD > Expand at Maximum > Save Webpage
  • Read each discussion page and extract requisite data
  • Generate a structured text file output in a predefined hierarchical form
  • Presenting data in form of comprehensive reports and dashboards
• Robust workflow supported auto-creation of data extraction patterns to expedite the data preparation process and improve data quality.

Quality Check and Audit:

• Programmable bots and macros were deployed to validate and verify extracted data.
• Alerts/notifications for missing or deleted comments were generated with ML backed algorithms.

Dispatch:

Automated upload of comprehensive output text file after conversion, transformation, and validation