fix: added the updated script existing index_datasets script to create the…

[Dhruvkumar-Microsoft]

… index

Purpose

This pull request updates the sample data processing scripts and the dataset indexing logic to support JSON and PDF files in addition to CSVs. The main improvements include unified and expanded blob upload steps, a new PDF text extraction capability, and streamlined indexing script execution. The changes also deprecate the separate PDF indexing script in favor of a single, consolidated indexing flow.

Blob upload and data ingestion improvements:

• The PowerShell (Process-Sample-Data.ps1) and Bash (process_sample_data.sh) scripts now upload both JSON and CSV files to blob storage, updating messages and error handling to reflect this broader support. [1] [2]
• The scripts now report success/failure for both CSV and JSON uploads together, simplifying the process. [1] [2]

Indexing and PDF support enhancements:

• The Python indexing script (index_datasets.py) now includes a function to extract text from PDF files using PyPDF2, handling encrypted/protected PDFs gracefully and returning clear status messages for unreadable or protected documents.
• The indexing logic in index_datasets.py is updated to process PDF files in addition to CSV and JSON, extracting their text content and including it in the indexed data.
• The PowerShell and Bash scripts now run only the unified dataset indexing script, deprecating the separate PDF indexing script (index_rfp_data.py). Related code for conditional execution of the old script is commented out. [1] [2]

Miscellaneous:

• The team config upload script now looks for RFP_Analysis_team.json instead of RFP_Analysis_team, ensuring consistency in file naming.

Does this introduce a breaking change?

• Yes
• No

How to Test

• Get the code

git clone [repo-address]
cd [repo-name]
git checkout [branch-name]
npm install

• Test the code

What to Check

Verify that the following are valid

• ...

Other Information