Prefect Structure
Directory Structure:
/.github
This directory is dedicated to the github actions workflows. The workflows sub-directory houses the workflow files that are written in yaml. There will always be two workflow files:prefect_trigger_deployment.yaml:This actions workflow file is responsible for the async functionality. It creates a deployment to trigger the main project deployment. This alows customer credentials to be passed into the main function.prefect_deployment.yaml:This action workflow file is responsible for the project deployment. It creates a deployment in prefect cloud upon a push to a certain branch.
/Endpoints
Any file within this directory are directly related to the project. This includes generic files, and unique project files.
/functions
This directory is for all of the generic files. This directory should be copied from project to project.
aws_functionality.py
This function is responsible for all functionality related to AWS. One example would be our upload_to_s3 function where the name is pretty self explanatory.
credential_management.py
This function is responsible for customer credential handling. It is not used in the deployment itself, but it is dependent upon the deployment trigger deployment. The functions contained in here are to get customer credentials, and refine them based on what api is requested.
prefect_trigger_function.py
This function is responsible for the async deployment. This will be the targeted file by the prefect_trigger_deployment.yaml file to create the prefect trigger deployment for a project. There are lines to this file that need to be modified from project to project. This includes the api that is being worked on, and the name of the deployment.
slack_notification.py
This function is our notification function. It sends slack notifications to certain slack channels based on project, and if it is an error or not. In addition, the get_run_logger functionality is tied into it as well to kill two birds with one stone.
All Other /Endpoint files
The rest of the files located within this directory are project specific. There will always be a main.py within this directory. This will be the main project file that is reference from within the prefect_deployment.yaml file.
/workflow-scripts
This directory is responsible for python scripts that help out the github actions workflows.
k8_infra_block.py
This file is responsible for creating/recreating the k8 job infrastructure block for the specified project. This will be triggered anytime the github actions workflows are triggered.
storage_block.py
This file is responsible for creating/recreating the project storage block. The storage block is set to github to utilize the default branch of a repository for the codebase reference. This is triggered anytime the github actions workflows are triggered.
k8s_flow_run_job_manifest.yaml
This file is responsible for the manifest that will be called to build the flow_run resources in prefect. It is necessary because the resource limit for these job kubernetes resources is at a low threshold. The threshold is sometimes below what it needs to be to complete a job. This manifest puts the resource limit at 8gb ram and 8vCPU's. This should be plenty of resources to complete any job we have to complete.*The only error may come with historical data sync*
Base Directory Breakdown
requirements.txt
This will be the file dedicated to the required python packages. It must exist in the base directory to be picked up by the helper scripts.
Generic Files:
Disclaimer
In all of the generic files, all of the notification pieces will need to be tailored to your particular project. In addition, ctrl+f < - > in order to find all areas that need mofications from project to project. This will only exist within the prefect_trigger_function.py.
Files:
Last updated