예제

• OpenShift AI에 Ray와 DeepSpeed를 가진 정밀한 Tune LLMs
• DreamBooth 및 Ray Train의 정밀한 Tune 안정 확산
• OpenShift AI에 Ray Tune와 Hyperparameters 최적화

통합 시험

# 필수품

• OpenShift 클러스터에 대한 관리자 액세스 (CRC는 괜찮습니다)

• OpenDataHub 또는 RHOAI 설치, 모든 분산 작업 부하 구성 요소를 활성화

• 설치 1.21

# 공통 환경 변수

• TEST_OUTPUT_DIR - 테스트 로그에 대한 출력 디렉토리
• TEST_TIMEOUT_SHORT - 짧은 작업에 대한 타임아웃 기간
• TEST_TIMEOUT_MEDIUM - 중간 작업을 위한 타임아웃 기간
• TEST_TIMEOUT_LONG - 긴 작업 시간 제한
• TEST_RAY_IMAGE (선택 사항) - 레이 클러스터 구성에 사용되는 레이 이미지
• MINIO_CLI_IMAGE (선택 사항) - /into s3 버킷에서 데이터를 업로드 / 다운로드하는 Minio CLI 이미지
• TEST_TIER (선택 사항) - 지정된 테스트 계층에 속하지 않는 테스트 계층을 실행하는 테스트 계층을 지정합니다. 지원된 시험 층: 연기, Tier1, Tier2, Tier3, 전위 및 포스트 급료. 시험 층은 또한 시험 모수 testTier를 사용하여 제공될 수 있습니다.

참고: quay.io/modh/ray:2.47.1-py312-cu128은 RayCluster 리소스를 만들기 위해 사용되는 기본 이미지입니다.

Distributed Workloads

Examples

• Fine-Tune LLMs with Ray and DeepSpeed on OpenShift AI
• Fine-Tune Stable Diffusion with DreamBooth and Ray Train
• Hyperparameters Optimization with Ray Tune on OpenShift AI

Integration Tests

Prerequisites

• Admin access to an OpenShift cluster (CRC is fine)

• Installed OpenDataHub or RHOAI, enabled all Distributed Workload components

• Installed Go 1.21

Common environment variables

• TEST_OUTPUT_DIR - Output directory for test logs

• TEST_TIMEOUT_SHORT - Timeout duration for short tasks

• TEST_TIMEOUT_MEDIUM - Timeout duration for medium tasks

• TEST_TIMEOUT_LONG - Timeout duration for long tasks

• TEST_RAY_IMAGE (Optional) - Ray image used for raycluster configuration

• MINIO_CLI_IMAGE (Optional) - Minio CLI image used for uploading/downloading data from/into s3 bucket

• TEST_TIER (Optional) - Specifies test tier to run, skipping tests which don’t belong to specified test tier. Supported test tiers: Smoke, Tier1, Tier2, Tier3, Pre-Upgrade and Post-Upgrade. Test tier can also be provided using test parameter testTier.

  NOTE: quay.io/modh/ray:2.47.1-py312-cu128 is the default image used for creating a RayCluster resource. If you have your own custom ray image which suits your purposes, specify it in TEST_RAY_IMAGE environment variable.

Environment variables for fms-hf-tuning test suite

• FMS_HF_TUNING_IMAGE - Image tag used in PyTorchJob CR for model training

Environment variables for fms-hf-tuning GPU test suite

• TEST_NAMESPACE_NAME (Optional) - Existing namespace where will the Training operator GPU tests be executed
• HF_TOKEN - HuggingFace token used to pull models which has limited access
• GPTQ_MODEL_PVC_NAME - Name of PersistenceVolumeClaim containing downloaded GPTQ models

To upload trained model into S3 compatible storage, use the environment variables mentioned below :

• AWS_DEFAULT_ENDPOINT - Storage bucket endpoint to upload trained dataset to, if set then test will upload model into s3 bucket
• AWS_ACCESS_KEY_ID - Storage bucket access key
• AWS_SECRET_ACCESS_KEY - Storage bucket secret key
• AWS_STORAGE_BUCKET - Storage bucket name
• AWS_STORAGE_BUCKET_MODEL_PATH (Optional) - Path in the storage bucket where trained model will be stored to

Environment variables for ODH integration test suite

• ODH_NAMESPACE - Namespace where ODH components are installed to
• NOTEBOOK_USER_NAME - Username of user used for running Workbench
• NOTEBOOK_USER_PASSWORD - Password of user used for running Workbench
• NOTEBOOK_USER_TOKEN - Login token of user used for running Workbench
• NOTEBOOK_IMAGE - Image used for running Workbench

To download MNIST training script datasets from S3 compatible storage, use the environment variables mentioned below :

• AWS_DEFAULT_ENDPOINT - Storage bucket endpoint from which to download MNIST datasets