feat: implement local folder upload support (#1771)#2317
Open
Saksham-official wants to merge 1 commit intoarc53:mainfrom
Open
feat: implement local folder upload support (#1771)#2317Saksham-official wants to merge 1 commit intoarc53:mainfrom
Saksham-official wants to merge 1 commit intoarc53:mainfrom
Conversation
Backend: accept multipart filenames that include relative paths (from webkitdirectory or recursive drag-drop), sanitize each path segment, reject basic path traversal attempts, and preserve directory structure when saving files. Zip handling updated to extract contents into a path that includes the zip's parent folder when present; non-extractable zips and non-zip files are saved using the safe relative path. Various variable renames and size enforcement calls adjusted to use the sanitized basename. Frontend: add a "Choose Folder" i18n string and UI button that uses a hidden webkitdirectory input; implement directory traversal for both the input and drag-and-drop (via webkitGetAsEntry) to collect files and set webkitRelativePath on File objects. Validate supported files using FILE_UPLOAD_ACCEPT_ATTR, append files to FormData with their relative paths so the server can reconstruct folder structure, and update related TypeScript typings and refs. These changes enable uploading folders while preserving structure and improve security and handling for extracted zip contents.
| @Saksham-official is attempting to deploy a commit to the Arc53 Team on Vercel. A member of the Team first needs to authorize it. |
Author
| @arc53 plz review this pr |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds the ability for users to upload entire local folders to DocsGPT, either via a new folder picker or by dragging and dropping folders into the upload area. This provides a more efficient workflow for ingesting structured documentation and large datasets.
Key Changes
Frontend (Upload.tsx):
Added a "Choose Folder" button that uses webkitdirectory for recursive folder selection.
Implemented a recursive directory traversal algorithm for the drag-and-drop area using webkitGetAsEntry, allowing users to simply drop folders to start ingestion.
Updated the upload logic to preserve and transmit relative paths, ensuring the backend knows the folder structure.
Backend (upload.py):
Enhanced the /api/upload endpoint to handle relative paths in file.filename.
Implemented a secure path-cleansing mechanism to safely recreate the directory hierarchy in temporary storage.
Integrated with the folder-aware ingestion logic introduced in #1770 to correctly map subdirectories in the knowledge base.
UX & Localization:
Added translation keys for folder selection.
Implemented automatic source naming based on the root folder name.