Dashboard
PHP:
8.2.30
OS:
Linux
User:
dh_nv8b2m
/
/
usr
/
lib
/
python3
/
dist-packages
/
sphinx
/
builders
/
__pycache__
📤 Upload
📝 New File
📁 New Folder
Close
Editing: linkcheck.cpython-310.pyc
o �$�amp � @ s d Z ddlZddlZddlZddlZddlZddlmZmZ ddlm Z ddl m Z ddl m Z ddlmZmZ ddlmZ dd lmZmZmZmZmZmZmZmZm Z m Z m Z dd l m Z m!Z!m"Z" dd l#m$Z$ dd l%m&Z& dd l'm(Z( ddl)m*Z*m+Z+m,Z, ddl-m.Z. ddl/m0Z0 ddl1m2Z2 ddl3m4Z4 ddl5m6Z6 ddl7m8Z8 ddl9m:Z: ddl;m<Z<m=Z=m'Z' ddl>m?Z?m@Z@mAZAmBZBmCZC ddlDmEZE e=�FeG�ZHe�Id�ZJeddeKfd eKfd eeL ff�ZMed d eNfd eeM ff�ZOed!deKfd eKfd eLfd"eKfd#eKfd$eLff�ZPed%d&eNfd eNff�ZQe eOe eNeKeKeLf f ZRd'd(iZSdZTd)ZUd*ZVd+e&d,eLfd-d.�ZWG d/d0� d0e �ZXd1e'j'j(d2eKd,eYfd3d4�ZZG d5d6� d6e0�Z[G d7d8� d8�Z\G d9d:� d:e�Z]G d;d<� d<e:�Z^d=e.deKd,eeK fd>d?�Z_d=e.d@e2d,dfdAdB�Z`d=e.d,eeKef fdCdD�ZadS )Ez� sphinx.builders.linkcheck ~~~~~~~~~~~~~~~~~~~~~~~~~ The CheckExternalLinksBuilder class. :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS. :license: BSD, see LICENSE for details. � N)�datetime�timezone)�parsedate_to_datetime)� HTMLParser)�path)� PriorityQueue�Queue)�Thread) �Any�Dict� Generator�List� NamedTuple�Optional�Pattern�Set�Tuple�Union�cast)�unquote�urlparse� urlunparse)�nodes)�Element)�Response)�ConnectionError� HTTPError�TooManyRedirects)�Sphinx)� DummyBuilder)�Config)�RemovedInSphinx50Warning)�BuildEnvironment)�__)�SphinxPostTransform)� encode_uri�logging�requests)�darkgray� darkgreen�purple�red� turquoise)� get_node_linez ([a-z]+:)?//� Hyperlink�uri�docname�lineno� CheckRequest� next_check� hyperlink� CheckResult�status�message�code� RateLimit�delay�Acceptz/text/html,application/xhtml+xml;q=0.9,*/*;q=0.8� g N@�node�returnc C s t jdtdd� t| �p dS )z� PriorityQueue items must be comparable. The line number is part of the tuple used by the PriorityQueue, keep an homogeneous type for comparison. z node_line_or_0() is deprecated.� �� stacklevelr )�warnings�warnr! r- )r= � rD �;/usr/lib/python3/dist-packages/sphinx/builders/linkcheck.py�node_line_or_0F s � rF c s@ e Zd ZdZdeddf� fdd� Zdededdfd d �Z� ZS ) �AnchorCheckParserz9Specialized HTML parser that looks for a specific anchor.� search_anchorr> Nc s t � �� || _d| _d S �NF)�super�__init__rH �found)�selfrH �� __class__rD rE rK S s zAnchorCheckParser.__init__�tag�attrsc C s0 |D ]\}}|dv r|| j krd| _ d S qd S )N)�id�nameT)rH rL )rM rP rQ �key�valuerD rD rE �handle_starttagY s ��z!AnchorCheckParser.handle_starttag) �__name__� __module__� __qualname__�__doc__�strrK r rV � __classcell__rD rD rN rE rG P s rG �response�anchorc C sP t |�}| jddd�D ]}t|t�r|�� }|�|� |jr nq |�� |jS )z�Reads HTML data from a response object `response` searching for `anchor`. Returns True if anchor was found, False otherwise. i T)� chunk_size�decode_unicode)rG � iter_content� isinstance�bytes�decode�feedrL �close)r] r^ �parser�chunkrD rD rE � check_anchor` s �ri c @ sl e Zd ZdZdZed�Zd.dd�Zede e fdd ��Z ede e e e f fd d ��Zede e fd d ��Zedee fdd��Zedeeef fdd��Zedeee eef f fdd��Zd.dd�Zdedee fdd�Zdede fdd�Z dede e fdd �Z dede fd d �Z d e!ddfd d!�Z"d"ed#ed$ed%ed&eddf d'd(�Z#d)e$ddfd*d+�Z%d.d,d-�Z&dS )/�CheckExternalLinksBuilderz+ Checks for broken external links. � linkcheckzCLook for any errors in the above output or in %(outdir)s/output.txtr> Nc C s8 i | _ t� | _i | _i | _t�d� t� | _t � | _ d S )Ng @) � hyperlinks�set�_good�_broken� _redirected�socket�setdefaulttimeoutr �_wqueuer �_rqueue�rM rD rD rE �initz s z CheckExternalLinksBuilder.initc C �. t jd| jjdf tdd� dd� | jjD �S )N�%s.%s is deprecated.�anchors_ignorer? r@ c S � g | ]}t �|��qS rD ��re�compile��.0�xrD rD rE � <listcomp>� � z<CheckExternalLinksBuilder.anchors_ignore.<locals>.<listcomp>)rB rC rO rW r! �config�linkcheck_anchors_ignoreru rD rD rE ry � � �z(CheckExternalLinksBuilder.anchors_ignorec C rw )Nrx �authr? r@ c S � g | ] \}}t �|�|f�qS rD r{ �r �pattern� auth_inforD rD rE r� � � z2CheckExternalLinksBuilder.auth.<locals>.<listcomp>)rB rC rO rW r! r� �linkcheck_authru rD rD rE r� � s ��z CheckExternalLinksBuilder.authc C rw )Nrx � to_ignorer? r@ c S rz rD r{ r~ rD rD rE r� � r� z7CheckExternalLinksBuilder.to_ignore.<locals>.<listcomp>)rB rC rO rW r! r� �linkcheck_ignoreru rD rD rE r� � r� z#CheckExternalLinksBuilder.to_ignorec C �" t jd| jjdf tdd� | jS )Nrx �goodr? r@ )rB rC rO rW r! rn ru rD rD rE r� � � �z CheckExternalLinksBuilder.goodc C r� )Nrx �brokenr? r@ )rB rC rO rW r! ro ru rD rD rE r� � r� z CheckExternalLinksBuilder.brokenc C r� )Nrx � redirectedr? r@ )rB rC rO rW r! rp ru rD rD rE r� � r� z$CheckExternalLinksBuilder.redirectedc C s t jd| jjdf tdd� d S )Nrx � check_threadr? r@ �rB rC rO rW r! ru rD rD rE r� � s �z&CheckExternalLinksBuilder.check_threadr] c C s: t jd| jjdf tdd� t| j| jd d i �}|�|�S )Nrx � limit_rater? r@ ) rB rC rO rW r! � HyperlinkAvailabilityCheckWorker�envr� r� )rM r] �workerrD rD rE r� � s � � z$CheckExternalLinksBuilder.limit_ratec C r� )Nrx �rqueuer? r@ )rB rC rO rW r! rt �rM r] rD rD rE r� � � �z CheckExternalLinksBuilder.rqueuec C s t jd| jjdf tdd� g S )Nrx �workersr? r@ r� r� rD rD rE r� � s �z!CheckExternalLinksBuilder.workersc C r� )Nrx �wqueuer? r@ )rB rC rO rW r! rs r� rD rD rE r� � r� z CheckExternalLinksBuilder.wqueue�resultc C s� | j �|jd �}t||j|j|j|j|jd�}| � |� |jdkr#d S |jdkr/|jdkr/d S |jr=t j d|j|jdd� |jdkrb|jrVt � t d �|j d |j � d S t � t d �|j � d S |jd krt � t d �|j � | � d |j||j|j� d S |jdkr�t � td �|j |j � d S |jdkr�| jjs�| jjr�t jtd�|j|j|j|jfd� nt � td�|j td|j � � | � d|j||j|jd |j � d S |jdk�rPzdtfdtfdtfdtfdtfd�|j \}}W n t�y dt}}Y nw ||d< | jj�r&t jd|j d | d |j |j|jfd� nt � |d�|j |d| d |j � � | � d | |j||j|jd |j � d S td |j ��) N)�filenamer1 r6 r8 r/ �info� unchecked�working�oldz(%16s: line %4d) T)�nonl�ignoredz -ignored- z: �localz -local- z ok r� zbroken link: %s (%s))�locationz broken z - r� � permanentlyz with Foundzwith See Other� temporarily)i- i. i/ i3 i4 zwith unknown code�textz redirect z to z redirected zUnknown status %s.)r� �doc2pathr0 �dictr1 r6 r8 r/ r7 �write_linkstat�loggerr� r( � write_entryr) �app�quiet�warningiserror�warningr# r+ r* r, �KeyErrorr� �linkcheck_allowed_redirects� ValueError)rM r� r� �linkstatr� �colorrD rD rE �process_result� sr � " �"� � �� � ���z(CheckExternalLinksBuilder.process_result�whatr0 r� �liner/ c C s | j �d||||f � d S )Nz%s:%s: [%s] %s )� txt_outfile�write)rM r� r0 r� r� r/ rD rD rE r� s z%CheckExternalLinksBuilder.write_entry�datac C s" | j �t�|�� | j �d� d S )N� )� json_outfiler� �json�dumps)rM r� rD rD rE r� " s z(CheckExternalLinksBuilder.write_linkstatc C s� t | j| j| �}t�d� tt�| jd�d��2| _ tt�| jd�d��| _ |� | j �D ]}| � |� q+W d � n1 s=w Y W d � n1 sLw Y | jrZd| j_d S d S )N� z output.txt�wz output.jsonr< )� HyperlinkAvailabilityCheckerr� r� r� r� �openr �join�outdirr� r� �checkrl r� ro r� � statuscode)rM �checkerr� rD rD rE �finish& s � � �� �z CheckExternalLinksBuilder.finish�r> N)'rW rX rY rZ rS r# �epilogrv �propertyr r ry r r r� r� r r[ r� r r� �intr� r� r r �floatr� r r� r r� r� r5 r� r� r� r� r� rD rD rD rE rj r s< 7� �rj c @ st e Zd Z ddedededdfdd�Zddd �Zdd d �Zd e e e f de e ddf fd d�Zde defdd�ZdS )r� Nr� r� �builderr> c C s^ || _ || _|| _i | _g | _dd� | jjD �| _|r%|j| _|j | _ d S t � | _t � | _ d S )Nc S rz rD r{ r~ rD rD rE r� ? r� z9HyperlinkAvailabilityChecker.__init__.<locals>.<listcomp>) r� r� r� � rate_limitsr� r� r� rt r� rs r� r r )rM r� r� r� rD rD rE rK 4 s z%HyperlinkAvailabilityChecker.__init__c C sH t | jj�D ]}t| j| j| j| j| j| j�}|� � | j � |� qd S �N) �ranger� �linkcheck_workersr� r� r� r� r� r� �startr� �append)rM �i�threadrD rD rE �invoke_threadsH s ��z+HyperlinkAvailabilityChecker.invoke_threadsc C s. | j �� | jD ] }| j �ttd �d� qd S rI )r� r� r� �putr2 �CHECK_IMMEDIATELY)rM r� rD rD rE �shutdown_threadsP s �z-HyperlinkAvailabilityChecker.shutdown_threadsrl c c s� � | � � d}|�� D ]$}| �|j�r!t|j|j|jddd�V q | j�t t |�d� |d7 }q d}||k rD| j � � V |d7 }||k s6| � � d S )Nr r� r� Fr< )r� �values�is_ignored_urir/ r5 r0 r1 r� r� r2 r� r� �getr� )rM rl � total_linksr4 �donerD rD rE r� U s � � � z"HyperlinkAvailabilityChecker.checkr/ c s t � fdd�| jD ��S )Nc 3 s � | ]}|� � �V qd S r� )�match)r �pat�r/ rD rE � <genexpr>i s � z>HyperlinkAvailabilityChecker.is_ignored_uri.<locals>.<genexpr>)�anyr� �rM r/ rD r� rE r� h s z+HyperlinkAvailabilityChecker.is_ignored_urir� r� )rW rX rY r" r rj rK r� r� r r[ r. r r5 r� �boolr� rD rD rD rE r� 3 s �� � $r� c sj e Zd ZdZ ddededededeee f de d df� fd d � Z dd d �Z de d ee fdd�Z� ZS )r� z;A worker class for checking the availability of hyperlinks.Nr� r� r� r� r� r� r> c s� || _ || _|| _|| _|| _dd� | j jD �| _dd� | j jD �| _|r2|j | _ |j | _ |j | _ n t � | _ i | _ i | _ t � jdd� d S )Nc S rz rD r{ r~ rD rD rE r� { s �z=HyperlinkAvailabilityCheckWorker.__init__.<locals>.<listcomp>c S r� rD r{ r� rD rD rE r� } r� T)�daemon)r� r� r� r� r� r� ry r� r� rn ro rp rm rJ rK )rM r� r� r� r� r� r� rN rD rE rK o s& �� z)HyperlinkAvailabilityCheckWorker.__init__c s� i ��j jr �j j�d< dtf��fdd� �dttttf f� �����fdd� �dtdtdtf�fd d � � d tdttttf f���fd d � } �j�� }z|\}��d u rZW d S �\�}}W n t yo |\}�}}Y nw �d u rvd S t ��j }z�j | j }W n ty� Y nw |t�� kr�t�t� �j�t|��d� �j�� qH||�\}}} |dkr�t�td�� td� � n �j�t�||||| �� �j�� qI)N�timeoutr> c sh t ��} d| j| jf d| j| jf �dg}|D ]}|� jjv r1tt�}|�� jj| � | S qi S )Nz%s://%sz%s://%s/�*)r �scheme�netlocr� �linkcheck_request_headersr� �DEFAULT_REQUEST_HEADERS�update)�url� candidates�u�headersr� rD rE �get_request_headers� s � �zAHyperlinkAvailabilityCheckWorker.run.<locals>.get_request_headersc s d�v r �� dd�\} }�jD ] }|�|�rd } nqn�} d }z| �d� W n ty4 t| �} Y nw �jD ] \}}|���rC nq8d }�� �d< zt|rv�jjrvt j | fd�j|d����}|� � t |t |��}|suttd�| ��nHzt j| fd�j|d����}|� � W n3 tttfy� } z$t|t�r�|jjd kr�� t j | fd�j|d����}|� � W Y d }~nd }~ww W n� t�y) } z]|jjd kr�W Y d }~d S |jjd k�r��|j�}|d ur��j�t|��d � W Y d }~d S dt|�dfW Y d }~S |jjdk�rdt|�dfW Y d }~S dt|�dfW Y d }~S d }~w t�yB } z dt|�dfW Y d }~S d }~ww t | �j } z�j | = W n t �yX Y nw |j �!d�| �!d�k�rgdS |j } |�rs| d| 7 } � | | ��r{dS |j"�r�|j"d j} d| | fS d| dfS )N�#r< �asciir� T)�streamr� r� zAnchor '%s' not found)�allow_redirectsr� r� i� i� )r� z - unauthorizedr F)� rate-limitedr� r r� r i� r� �/�r� r� r ���r� )#�splitry r� �encode� UnicodeErrorr% r� r� �linkcheck_anchorsr' r� �raise_for_statusri r � Exceptionr# �headr r r rb r] � status_coder� r� r� r2 r[ r r� r� r� r� �rstrip�history) �req_urlr^ �rexr� r� r] rL �errr3 r� �new_urlr8 )�allowed_redirectr� r4 �kwargsrM r/ rD rE � check_uri� s� �� � � ��� �� ����� ��� � z7HyperlinkAvailabilityCheckWorker.run.<locals>.check_urir� r c s4 � j j�� D ]\}}|�| �r|�|�r dS qdS )NTF)r� r� �itemsr� )r� r �from_url�to_urlru rD rE r s �z>HyperlinkAvailabilityCheckWorker.run.<locals>.allowed_redirectr0 c s, t ��dks ��d�r dS ��d�s4t���rdS t��j�| ��}t�t� |���r-dS d�j �< dS ��j v r;dS ��j v rHd �j � dfS ��j v r\d �j � d �j � d fS t �jj�D ]}� � \}}}|d krp nqb|d kr|�j ��� n|d kr�|�j �< n |d kr�||f�j �< |||fS ) Nr )r� zmailto:ztel:)r� r� r )zhttp:zhttps:r� r� )r� r� r )r� r� r r� r� r< r� )�len� startswith�uri_rer� r �dirnamer� r� �existsr� ro rn rp r� r� �linkcheck_retries�add)r0 �srcdir�_r6 r� r8 )r rM r/ rD rE r� s8 � z3HyperlinkAvailabilityCheckWorker.run.<locals>.checkTFr� z-rate limited- z | sleeping...)r� �linkcheck_timeoutr r r[ r� r� r� r� r� r r� r� r3 r� �time�sleep�QUEUE_POLL_SECSr� r2 � task_doner� r� r( r� r5 ) rM r� � check_requestr3 r0 r1 r� r6 r� r8 rD )r r r� r4 r rM r/ rE �run� sH &`$% � � �z$HyperlinkAvailabilityCheckWorker.runr] c C s d }|j �d�}|rAzt|�}W n* ty: zt|�}W n ttfy( Y nw t�|�}|t�t j � � � }Y nw t � � | }t |j�j}|d u r| jj}z| j| }W n tya t}Y nw |j} d| }||krs| |k rs|}||kryd S t � � | }t||�| j|< |S )Nz Retry-Afterg @)r� r� r� r� r � TypeErrorr � timestamp�nowr �utc� total_secondsr r r� r� r� � linkcheck_rate_limit_timeoutr� r� � DEFAULT_DELAYr: r9 ) rM r] r3 � retry_afterr: �untilr� � max_delay� rate_limit�last_wait_timerD rD rE r� R sB � �� � z+HyperlinkAvailabilityCheckWorker.limit_rater� r� )rW rX rY rZ r" r r r r[ r9 rj rK r$ r r r� r� r\ rD rD rN rE r� l s �� ��� Fr� c @ s& e Zd ZdZdZdeddfdd�ZdS )�HyperlinkCollector)rk � r r> Nc K s� t t| jj�}|j}| j�tj�D ]*}d|vrq|d }| j� d|�}|r'|}t |�}t || j j |�}||vr;|||< q| j�tj�D ].} | d �d�}|rqd|v rq| j� d|�}|r]|}t | �}t || j j |�}||vrq|||< qCd S )N�refuri�linkcheck-process-urir� �?z://)r rj r� r� rl �document�traverser � reference�emit_firstresultr- r. r� r0 �imager� ) rM r r� rl �refnoder/ �newurir1 �uri_info�imgnoderD rD rE r$ } s4 � ��zHyperlinkCollector.run)rW rX rY �builders�default_priorityr r$ rD rD rD rE r1 y s r1 r� c C sH t |�}|jdkr"|jr"|j�d�}|s"d|j� �}t|j|d��S dS )z�Rewrite anchor name of the hyperlink to github.com The hyperlink anchors in github.com are dynamically generated. This rewrites them before checking and makes them comparable. z github.comz user-content-)�fragmentN)r �hostnamerA r r �_replace)r� r/ �parsed�prefixedrA rD rD rE �rewrite_github_anchor� s rF r� c C s� t | jj�� �D ]B\}}z6zt�|�| jjt�|�< W n tjy9 } zt�t d�|j |j � W Y d}~nd}~ww W | jj� |� q| jj� |� w dS )zFCompile patterns in linkcheck_allowed_redirects to the regexp objects.z=Failed to compile regex in linkcheck_allowed_redirects: %r %sN) �listr� r� r r| r} �errorr� r� r# r� �msg�pop)r� r� r� r� �excrD rD rE �#compile_linkcheck_allowed_redirects� s ���� �rL c C s� | � t� | �t� | �dg d � | �di d � | �dg d � | �di d � | �ddd � | �dd d tg� | �dd d � | �d d d � | �d d gd � | �ddd � | �d� | jdtdd� dd d d�S )Nr� r� r� r� r r< r r� � r Tr� z^!r* g �r@r4 z config-initedr2 )�priority�builtin)�version�parallel_read_safe�parallel_write_safe) � add_builderrj �add_post_transformr1 �add_config_valuer� � add_event�connectrL )r� rD rD rE �setup� s$ �rX )brZ r� r| rq r rB r r � email.utilsr � html.parserr �osr �queuer r � threadingr �typingr r r r r r r r r r r � urllib.parser r r �docutilsr �docutils.nodesr r' r �requests.exceptionsr r r �sphinx.applicationr �sphinx.builders.dummyr � sphinx.configr �sphinx.deprecationr! �sphinx.environmentr" � sphinx.localer# �!sphinx.transforms.post_transformsr$ � sphinx.utilr% r& �sphinx.util.consoler( r) r* r+ r, �sphinx.util.nodesr- � getLoggerrW r� r} r r[ r� r. r� r2 r5 r9 �CheckRequestTyper� r� r! r+ rF rG r� ri rj r� r� r1 rF rL rX rD rD rD rE �<module> s� 4 � � �� B9 $
Save
Cancel