feat: clear free plan workflow run logs

[hjlarry]

Important

1. Make sure you have read our contribution guidelines
2. Ensure there is an associated issue and you have been assigned to it
3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

fix #29776

related PR:
#29842
#29829
#29638

Todo:
rm api/services/clear_free_plan_tenant_expired_logs.py in the future.

Usage

1. update db

to use the created_at index of workflow_runs table

flask db upgrade 

2. dry run

flask clean-workflow-runs --start-from="2024-01-01" --end-before="2024-01-31" --dry-run 

then you can get some logs like:

... Dry run complete. Would delete 2024 workflow runs between 2014-01-01T00:00:00 and 2024-01-02T00:00:00; related records: node_executions 2022, offloads 6066, app_logs 2024, trigger_logs 2022, pauses 2022, pause_reasons 2022 

you can compare with the sql result (maybe slow):

WITH params AS ( SELECT TIMESTAMP '2024-01-01 00:00:00' AS start_after, TIMESTAMP '2024-01-31 00:00:00' AS end_before ) SELECT 'workflow_runs' AS table_name, COUNT(*) AS record_count FROM workflow_runs, params WHERE created_at >= start_after AND created_at < end_before UNION ALL SELECT 'workflow_node_executions', COUNT(*) FROM workflow_node_executions, params WHERE created_at >= start_after AND created_at < end_before UNION ALL SELECT 'workflow_node_execution_offload', COUNT(*) FROM workflow_node_execution_offload, params WHERE created_at >= start_after AND created_at < end_before UNION ALL SELECT 'workflow_app_logs', COUNT(*) FROM workflow_app_logs, params WHERE created_at >= start_after AND created_at < end_before UNION ALL SELECT 'workflow_trigger_logs', COUNT(*) FROM workflow_trigger_logs, params WHERE created_at >= start_after AND created_at < end_before UNION ALL SELECT 'workflow_pauses', COUNT(*) FROM workflow_pauses, params WHERE created_at >= start_after AND created_at < end_before UNION ALL SELECT 'workflow_pause_reasons', COUNT(*) FROM workflow_pause_reasons, params WHERE created_at >= start_after AND created_at < end_before ORDER BY table_name; 

4. run

To clear run logs within a specified time range.

flask clean-workflow-runs --start-from="2024-01-01" --end-before="2024-01-31" 

To clear the run logs before 90 days:

flask clean-workflow-runs --days=90 

To schedule run the clear task:

ENABLE_WORKFLOW_RUN_CLEANUP_TASK=true 

Screenshots

Before	After
...	...

Checklist

• This change requires a documentation update, included: Dify Document
• I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
• I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
• I've updated the documentation accordingly.
• I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

[gemini-code-assist]

Code Review

This pull request introduces a new feature to clean up expired workflow run logs for free-tier tenants. It includes a new CLI command, a scheduled Celery task, a database migration to add a necessary index, and the core service logic for performing the cleanup in batches. The implementation is robust, with good test coverage for the new components. My review includes a suggestion to optimize the database deletion logic for better performance and memory efficiency.

[hj24]

except the migration version, others LGTM

[hj24]

LGTM