Dashboard
PHP:
8.2.30
OS:
Linux
User:
dh_nv8b2m
/
/
usr
/
lib
/
python3
/
dist-packages
/
awscli
/
customizations
/
emr
/
__pycache__
📤 Upload
📝 New File
📁 New Folder
Close
Editing: createcluster.cpython-310.pyc
o .&�a'e � @ s� d dl Z d dlmZ d dlmZ d dlmZ d dlmZ d dlmZ d dlmZ d dlm Z d d lm Z d d lm Z d d lm Z d d lm Z d d lmZ d dlmZ d dlmZ d dlmZ d dlmZ G dd� de�ZdS )� N)� BasicCommand)�applicationutils)�argumentschema)� constants)� emrfsutils)�emrutils)� exceptions)� hbaseutils)�helptext)�instancegroupsutils)�instancefleetsutils)� steputils)�Command)� EC2_ROLE_NAME)� EMR_ROLE_NAME)�jsonc @ s� e Zd ZdZejZg dejd��dejd��de j ej d��dej d��dej d��d d d ejd ��d d d d��de jejd��ddejd��dejd��dejd��dejd��dejd��dd ejd��dejd��deje jd��d d d ejd ��d d d d��d ejd��d d d!ej d ��d"d d!d��d#d d$ej d ��d%d d$d��d&d'ej e j d(��d)ej e j!d��d*ej"e j#d��d+ej$e j%d��d,e j&ej'd��d-ej(d��d.e j)ej*d��d/ej+d��d0ej,d��d1ej-d��d2ej.d��d3e j/ej0d��d4d5ej1d6��d7e j2ej3d��d8e j4ej5d��d9e j6ej7d��Z8e9�:d:d;�Z;e9�:d:d<�Z<d=d>� Z=d?d@� Z>dAdB� Z?dCdD� Z@dEdF� ZAdGdH� ZBdIdJ� ZCdKdL� ZDdMdN� ZEdOdP� ZFdQdR� ZGdSdT� ZHdUS )V� CreateClusterzcreate-clusterz release-label)�name� help_textz ami-versionzinstance-groups)r �schemar z instance-typezinstance-countzauto-terminate� store_true�auto_terminate)r �action� group_namer zno-auto-terminate)r r r zinstance-fleetsr zDevelopment Cluster)r �defaultr zlog-urizlog-encryption-kms-key-idz service-rolezauto-scaling-rolezuse-default-roles)r r r �configurationszec2-attributes)r r r ztermination-protected�termination_protectedzno-termination-protectedzscale-down-behaviorzvisible-to-all-users� visibilityzno-visible-to-all-userszenable-debugging�debugzno-enable-debugging�tags�+)r �nargsr r zbootstrap-actions� applications�emrfs�stepszadditional-infozrestore-from-hbase-backupzsecurity-configurationz custom-ami-idzebs-root-volume-sizezrepo-upgrade-on-bootzkerberos-attributeszstep-concurrency-level�integer)r � cli_type_namer zmanaged-scaling-policyzplacement-group-configszauto-termination-policy�emrzcreate-cluster-synopsis.txtzcreate-cluster-examples.rstc C s� i }|j |d< | �|� d}|jdu r |jd ur tjdd|d��|jdu r7|jd ur7d|jv r7tjdd|d��|jd urH|jd urHtjd d d ��i }|jd urXt � |j�|d < n t j |j|j |jd �|d< |jd ur�|j|d< |jd ur�z t�|j�|d< W n ty� td��w |jd u r�|jd ur�t�d|j�}|d u r�tj|jd��|j|d< t�|d|j� t�|d|j� |j d ur�t�|d|j � |jdu r�t |_|jd u r�i |_t |jd< t�|d|j� |jd u�r|d D ]}d|� � v �r|j d u �rt�!� �q�t�|d|j � |j"d u�rt�|d|j"� |j#d u �r(|j$d u �r(d|_#t�%|j#d |j$d �|d < t�%|j&d |j'd!�|d"< |j(d u �rO|j)d u �rOd|_(t�%|j(d#|j)d$�|d%< t�*|j+�|d&< ||d'< |jd u�ru| j,||jd(� t�%|j-d)|j.d*�}|jd u �r�|du �r�tj/�|du �r�| j0|d+| �1||�gd,� |j2d u�r�|jd u �r�t3j4| j5|j2|d d-�\} } } | �0|d.| � | �0|d/| � | �0|d+| � ng |d0< |j2D ] } |d0 �6| � �q�|j7} | d u�r t8j9| �:d1�| �:d2�d3�}tj;t<j=t<j>t<j?|d4�}| �0|d+|g� |j@d u�r| jA||j@d5� |jBd u�r)| jC||jB|jd6� |jDd u�rBtEjF|jD| j5|jd7�}| j0|d+|d,� |jGd u�rPt�|d8|jG� |jHd u�r^t�|d9|jH� |jId u�rnt�|d:tJ|jI�� |jKd u�r|t�|d;|jK� |jLd u�r�t�|d<|jL� |jMd u�r�|jM|d=< |jNd u�r�t�|d>|jN� |jOd u�r�t�|d?|jO� |jPd u�r�t�|d@|jP� | �Q|� t�R| jSdA|| j5|jT|jU�}| �V|�}t�W| jSdA||� dBS )CN�Namez| Either choose --use-default-roles or use both --service-role <roleName> and --ec2-attributes InstanceProfile=<profileName>.Tz--use-default-rolesz--service-role)�option1�option2�message�InstanceProfilez --ec2-attributes InstanceProfilez--instance-groupsz--instance-fleets�r) r* �InstanceFleets)�instance_groups� instance_type�instance_count�InstanceGroups� ReleaseLabel�Configurationsz=aws: error: invalid json argument for option --configurationsz\d?\..*)� ami_version� AmiVersion�AdditionalInfo�LogUri�LogEncryptionKmsKeyId� ServiceRole�AutoScalingPolicy�AutoScalingRole�ScaleDownBehaviorFz--no-auto-terminatez--auto-terminate�KeepJobFlowAliveWhenNoStepsz--termination-protectedz--no-termination-protected�TerminationProtectedz--visible-to-all-usersz--no-visible-to-all-users�VisibleToAllUsers�Tags� Instances)�cluster� parsed_attrsz--enable-debuggingz--no-enable-debugging�Steps�rC �key�value)�region�parsed_applicationsr5 �NewSupportedProducts�BootstrapActions� Applications�Dir� BackupVersion)�dir�backup_version)�jarr �action_on_failure�args)rC �parsed_boostrap_actions)rC � emrfs_args� release_label)�parsed_step_listrI rW �SecurityConfiguration� CustomAmiId�EbsRootVolumeSize�RepoUpgradeOnBoot�KerberosAttributes�StepConcurrencyLevel�ManagedScalingPolicy�PlacementGroupConfigs�AutoTerminationPolicy� run_job_flowr )Xr �#_validate_release_label_ami_version�use_default_roles� service_roler �MutualExclusiveOptionError�ec2_attributesr/ �instance_fleetsr �"validate_and_build_instance_fleetsr �"validate_and_build_instance_groupsr0 r1 rW r r �loads� ValueErrorr5 �re�match�InvalidAmiVersionErrorr � apply_dict�additional_info�log_uri�log_encryption_kms_key_idr r �keys�auto_scaling_role�MissingAutoScalingRoleError�scale_down_behavior�no_auto_terminater �apply_boolean_optionsr �no_termination_protected�visible_to_all_users�no_visible_to_all_users� parse_tagsr �_build_ec2_attributes�enable_debugging�no_enable_debugging� LogUriError�_update_cluster_dict�_build_enable_debuggingr" r �build_applicationsrI �append�restore_from_hbase_backupr �$build_hbase_restore_from_backup_args�get� build_stepr �HBASE_JAR_PATH�HBASE_RESTORE_STEP_NAME�CANCEL_AND_WAIT�bootstrap_actions�_build_bootstrap_actionsr# �_handle_emrfs_parametersr$ r �build_step_config_list�security_configuration� custom_ami_id�ebs_root_volume_size�int�repo_upgrade_on_boot�kerberos_attributes�step_concurrency_level�managed_scaling_policy�placement_group_configs�auto_termination_policy� _validate_required_applications�call�_session� endpoint_url� verify_ssl�_construct_result�display_response)�self� parsed_args�parsed_globals�params� service_role_validation_message�instances_config�is_valid_ami_version�instance_group�debugging_enabled�app_list�ba_list� step_list� application�hbase_restore_configrT � step_config� steps_list�run_job_flow_response�constructed_result� r� �I/usr/lib/python3/dist-packages/awscli/customizations/emr/createcluster.py�_run_main_command� s� � � � � ���� � � � � � � ���� �� �� �� ����� ��� � � �� � � � � � � � � � �z CreateCluster._run_main_commandc C s: d }d }|d ur|� d�}|� d�}|d ur||d�S i S )N� JobFlowId� ClusterArn)� ClusterIdr� )r� )r� �run_job_flow_result� jobFlowId� clusterArnr� r� r� r� { s �z CreateCluster._construct_resultc C sx |� � }|d }d|v rd|v rtjddd��d|v r&d|v r&tjddd��d|v s.d|v r9d|v s6d|v r9tj�tj|d|dd � tj|d|d d � tj|d|d d � d|v rht� |d < tj|d|d dd � d|v r|t� |d < tj|d|d dd � tj|d |dd � tj|d|dd � tj|d|dd � tj|d|dd � tj|d|dd � tj|d|dd � tj|d|d� |S )NrB �SubnetId� SubnetIdsr- �AvailabilityZone�AvailabilityZones�KeyName� Ec2KeyName�� src_params�src_key� dest_params�dest_key� Ec2SubnetId� Ec2SubnetIds� Placementr, � JobFlowRole� EmrManagedMasterSecurityGroup� EmrManagedSlaveSecurityGroup�ServiceAccessSecurityGroup� AdditionalMasterSecurityGroups� AdditionalSlaveSecurityGroups�r� rG rH )rt r rf �SubnetAndAzValidationErrorr � apply_params�dict�apply)r� rC rD rt � instancesr� r� r� r~ � s� ����� � �������z#CreateCluster._build_ec2_attributesc C s� |� d�}|d u r g }g }t|�t|� tjkr td��|D ]9}i }|� d�d ur1|� d�|d< ntj|d< i }tj|d|dd� tj|d|dd� tj|d|d� |� |� q || }t|�d krf||d< |S ) NrL zGaws: error: maximum number of bootstrap actions for a cluster exceeded.r( �Pathr� �Args�ScriptBootstrapActionr� r ) r� �lenr �MAX_BOOTSTRAP_ACTION_NUMBERrl �BOOTSTRAP_ACTION_NAMEr r� r� r� ) r� rC rU �cluster_ba_listr� �ba� ba_config�script_arg_config�resultr� r� r� r� � s@ � ��� z&CreateCluster._build_bootstrap_actionsc C sL |j r tj}tjg}nt�| j�}tjtj| jd�g}tj tj tj ||d�S )N)� relative_pathrI )r rS rR rT ) rW r �COMMAND_RUNNER�DEBUGGING_COMMANDr �get_script_runnerrI � build_s3_link�DEBUGGING_PATHr� �DEBUGGING_NAME�TERMINATE_CLUSTER)r� r� r� rR rT r� r� r� r� � s ��z%CreateCluster._build_enable_debuggingc C s@ ||� � v r|| |7 < |S |d ur t|�dkr |||< |S )Nr )rt r� )r� rC rG rH r� r� r� r� s �z"CreateCluster._update_cluster_dictc C sL |j d ur|jd urtjddd��|j d u r"|jd u r$tjddd��d S d S )Nz --ami-versionz--release-labelr- )r5 rW r rf �RequiredOptionsError)r� r� r� r� r� rc s � ��z1CreateCluster._validate_release_label_ami_versionc C sr t g �}|jd urt dd� |jD ��}| �||�}|jd ur+tj|vr+|�tj�� � t|�dkr7t j |d��d S )Nc S s g | ]}|d � � �qS )r( )�lower)�.0�appr� r� r� � <listcomp> s zACreateCluster._validate_required_applications.<locals>.<listcomp>r )r" ) �setr" �#_get_missing_applications_for_stepsr� r �HBASE�add�titler� r �MissingApplicationsError)r� r� �specified_apps� missing_appsr� r� r� r� s �� ��z-CreateCluster._validate_required_applicationsc C s� t tjtjtjg�}t g �}|jd urB|jD ]+}t|�t|�kr# |S |�d�}|d urA|�� }||v rA||vrA|� |d � � � q|S )N�Type) r� r �HIVE�PIG�IMPALAr$ r� r� r� r� r� )r� r� r� �allowed_app_stepsr� �step� step_typer� r� r� r� + s" � ��z1CreateCluster._get_missing_applications_for_stepsc C s |j r dd� |D �}|S )Nc S s$ g | ]}|j d kr|j dkr|�qS )re �instance_profile)r )r� �xr� r� r� r� ? s �zICreateCluster._filter_configurations_in_special_cases.<locals>.<listcomp>)rd )r� r r� �parsed_configsr� r� r� �'_filter_configurations_in_special_cases< s z5CreateCluster._filter_configurations_in_special_casesc C sP |r| � |� t�|�}| j|d|gd� d S t�| j|�}| j|d|d� d S )Nr4 rF rL )� validate_no_emrfs_configurationr �build_emrfs_confiurationr� � build_bootstrap_action_configsrI )r� rC rV rW �emrfs_configuration�emrfs_ba_config_listr� r� r� r� D s � �� �z&CreateCluster._handle_emrfs_parametersc C s< d|v r|d D ]}|d ur|� d�tjkrtj�qd S d S )Nr4 �Classification)r� r � EMRFS_SITEr � DuplicateEmrFsConfigurationError)r� rC �configr� r� r� r T s ��z-CreateCluster.validate_no_emrfs_configurationN)I�__name__� __module__� __qualname__�NAMEr �CREATE_CLUSTER_DESCRIPTION� DESCRIPTION� RELEASE_LABEL� AMI_VERSIONr �INSTANCE_GROUPS_SCHEMA�INSTANCE_GROUPS� INSTANCE_TYPE�INSTANCE_COUNT�AUTO_TERMINATE�INSTANCE_FLEETS_SCHEMA�INSTANCE_FLEETS� CLUSTER_NAME�LOG_URI�LOG_ENCRYPTION_KMS_KEY_ID� SERVICE_ROLE�AUTOSCALING_ROLE�USE_DEFAULT_ROLES�CONFIGURATIONS�EC2_ATTRIBUTES�EC2_ATTRIBUTES_SCHEMA�TERMINATION_PROTECTED�SCALE_DOWN_BEHAVIOR� VISIBILITY� DEBUGGING�TAGS� TAGS_SCHEMA�BOOTSTRAP_ACTIONS�BOOTSTRAP_ACTIONS_SCHEMA� APPLICATIONS�APPLICATIONS_SCHEMA�EMR_FS� EMR_FS_SCHEMA� STEPS_SCHEMA�STEPS�ADDITIONAL_INFO� HBASE_RESTORE_FROM_BACKUP_SCHEMA�RESTORE_FROM_HBASE�SECURITY_CONFIG� CUSTOM_AMI_ID�EBS_ROOT_VOLUME_SIZE�REPO_UPGRADE_ON_BOOT�KERBEROS_ATTRIBUTES_SCHEMA�KERBEROS_ATTRIBUTES�STEP_CONCURRENCY_LEVEL� MANAGED_SCALING_POLICY_SCHEMA�MANAGED_SCALING_POLICY� PLACEMENT_GROUP_CONFIGS_SCHEMA�PLACEMENT_GROUP_CONFIGS� AUTO_TERMINATION_POLICY_SCHEMA�AUTO_TERMINATION_POLICY� ARG_TABLEr � FROM_FILE�SYNOPSIS�EXAMPLESr� r� r~ r� r� r� rc r� r� r r� r r� r� r� r� r ! s� �������� �� �������������� �� ��!��#��&��)��+��-��0��2��5��7��:��=��@��C��F��H��K��M��O��Q��S��V��Y��\��_�� c r F% r )rm � awscli.customizations.commandsr �awscli.customizations.emrr r r r r r r r r r r �!awscli.customizations.emr.commandr �#awscli.customizations.emr.constantsr r �botocore.compatr r r� r� r� r� �<module> s$
Save
Cancel